Data cloned on F5 and sent to Cribl workers
Posting this to get community feedback and track our progress in case anyone encounters this issue in the future.
We are attempting to use the data cloning functionality on our F5 load balancer to duplicate data to our Cribl workers. We want to do this so that we can route, filter, and reduce production data in Cribl without impacting production operations. When the time comes, Cribl will be ready for the cutover without much impact at all, hopefully.
Right now, the F5 is sending data to our syslog servers, which then send to Splunk. The end goal is to replace syslog with Cribl, but we want to be able to see all the data in Cribl before then so that we can operate on it. We are successfully cloning data that comes from Splunk UFs and HFs, but the cloned syslog data from the F5 is not being seen by the Cribl worker.
The specific problem is this: The data is making it to the to the correct interface and port on the worker, but is not being picked up by Cribl for whatever reason.
I have verified that data is coming in by using tcpdump to view the packets. We have turned off the local firewall on the server. There are no iptables rules dropping the traffic. I have verified "net.ipv4.all.rp_filter" is not present in sysctl.
I'm not an F5 engineer, so I am mostly looking for feedback from that standpoint. Could the F5 be sending data in the wrong format, if that makes sense? My logic is that if the data is hitting the correct interface/port on the worker, then Cribl should be picking it up, but maybe that logic is flawed. Maybe there is a Linux configuration I am missing.
Funny thing is, we can see the UDP port monitoring traffic that the F5 sends to the worker, but not the data that it is cloning and sending.
Comments
-
Make sure ports match up and cribl is listening on the interface the events are arriving on.
Is the data coming as UDP or TCP? tcpdump will tell you, if it's UDP I think a syslog source is your only option. If TCP, for debugging purposes I'd start with a raw TCP source, so you can see exactly what it looks like.
0 -
Good advice. And, good news! Raw UDP was added as a source recently, allowing capture of any arbitrary UDP-based data.
Most often when I'm missing data in a situation like this, I check firewall settings, both on the network and the linux system's local firewall (eg, iptables).
0
