How can I troubleshoot a cribl destination (Splunk HEC) not sending data?
Hello. How can I troubleshoot a cribl destination (Splunk HEC) not sending data? destination is seen live/green check, but no data flowing per tcpdump on source or destination. this goes out through a router and same data arriving in the main index correctly but want in a separate one. curl from cribl to splunk works (`curl -vk https://172.16.x.y:8088/services/collector/event -H 'Authorization: Splunk TOKEN' -d '{"event":"test"}'`) Thanks
Answers
-
my first step would be to send to `nc` running on a local host and interrogate the actual payload
0 -
Trying that but I suspect that I will get nothing. Destination out traffic charts are at 0. Not sure if rules order matter in router destination but tried to change those as I have few disabled one/set false.
0 -
For now, not getting expected data. If using destination Test option, I get it in tcpdump and splunk. I would think the problem is likely not in destination but before. But not sure why as the main routing is working and not this one.
0 -
Running the test from within the HEC destination doesn't involve routing in any way
0 -
yes and this part is working. so that means destination is configured fine, right? in this case, problem is earlier
0 -
strangely after switching back localhost to splunk system, now, I have got data in tcdpump 8088 but strange correct/incorrect marking... all in aws network
0 -
but no data in splunk UI
0 -
the test function should produce data in splunk, if you specify the correct index
0 -
fix that first. then tackle routing config
0 -
Ok. got it. that was the router "final" option check. now got it in splunk
0 -
Thanks
0
