Every firewall event is being cloned
I'm trying to figure out why every event from my pfsense firewall is being cloned as it passes through the pipeline. When I look at Preview Full from the Pipeline, I see the first event in my sample data file being processed as desired through the pipeline, then the second event is an unprocessed clone of the first. The third event is processed correctly by the pipeline, and the fourth event is an unprocessed clone, etc. When enabling the Advanced Setting to Show Internal Fields, I see a field called "__cloneCount". The value of this field in the correctly processed events is, "__cloneCount: 0", and the value for the unprocessed, cloned events is, "__cloneCount: 1".
I've tried adding a Drop function to the end of the pipeline to drop events where, "__cloneCount == 1", but this has no effect. I've also tried to use Chain to call another pipeline to Drop the clone events, but this also has no effect. I though about adding a post-processing pipeline to the destination, but I'm sending to DevNull and it looks like you can't do post-processing pipelines here.
I'm pretty stuck and I can't find a lot of documentation on the __cloneCount internal field. Any pointers are appreciated.
Best Answer
-
@Bryan Rapp thanks for sharing your routes and pipelines with me. Here's what's going on… Because your Pfsense route is configured as
final=false, the matching events are send through the configured pipeline and to the destination, but they also continue down the route list looking for additional matches.Because there's no additional matches, what you're seeing is "cloned" events hitting the default route at the bottom of the route table.
I'd recommend you enable the "final" flag on the Pfsense route so you don't see these duplicate events.
Alternately, you could add an explicit "default" route with a devnull destination in your route table as the last entry if you wanted.
Some recommended reading:
This is what the Cribl default route entry looks like in my lab as an example. I also have an explicit "default" route added (entry #6).
0
Answers
-
__cloneCount is the result of a Clone function. Does your pipeline use Clone? Is there a pre-processing Pipeline defined in your source that uses Clone?
0 -
Double check you didn't accidentally click the "Add Clone" button on the route.
0 -
The pipeline doe not include a Clone function. There is also no pre-processing Pipeline attached to the syslog source.
0 -
Good thinking, but unfortunately this was not the case. I recreated the route too just for good measure, and I'm still seeing the same behavior.
0 -
Would you be able to DM me your pipeline so I can review?
0 -
Sure thing
0 -
@Bryan Rapp thanks for sharing your routes and pipelines with me. Here's what's going on… Because your Pfsense route is configured as
final=false, the matching events are send through the configured pipeline and to the destination, but they also continue down the route list looking for additional matches.Because there's no additional matches, what you're seeing is "cloned" events hitting the default route at the bottom of the route table.
I'd recommend you enable the "final" flag on the Pfsense route so you don't see these duplicate events.
Alternately, you could add an explicit "default" route with a devnull destination in your route table as the last entry if you wanted.
Some recommended reading:
This is what the Cribl default route entry looks like in my lab as an example. I also have an explicit "default" route added (entry #6).
0 -
Thanks, Brendan! Setting the Final flag on the Pfsense route solved the problem. Everything looks as it should. Thanks again for your help.
0
