Breaking a single stream input into separate pipelines

Draco3

I have a data stream that contains various different log types. For example, I have a Windows log, a firewall log and an antivirus log in the same stream. Each of the events from the source is tagged for example

type: windows
type: firewall
type: antivirus

What I want to do is split these logs by type. Essentially I would like to do:

• If the type equals windows then drop everything else and run it through a pipeline / route to transform my data with a windows parser and ship my data

• If the type equals firewall then drop everything else and run it through a pipeline / route to transform my data with a firewall parser and ship my data

• If the type equals antivirus then drop everything else and run it through a pipeline / route to transform my data with a antivirus parser and ship my data

I started thinking that I would use the eval function to filter based on this and group them together but unsure if this would destroy my data due to order. I also started thinking that I could do some pre-processing pipelines on the source, but the source only allows 1 pipeline to select (not multiple). If my pipeline was configured as follows:

Group A

1. Eval | filter type===‘windows’
2. Drop | filter type!=‘windows’
3. Chain | windows_pipeline

Group B

1. Eval | filter type===‘firewall’
2. Drop | filter type!=‘firewall’
3. Chain | firewall_pipeline

Group C

1. Eval | filter type===‘antivirus’
2. Drop | filter type!=‘antivirus’
3. Chain | antivirus_pipeline

However if I do this I think Group B and Group C would fail Group A dropped that data. I considered Event Breakers, but this doesn’t seem to fit what Event Breakers are for.

Any help would be appreciated. Thanks.

Brendan Dalpe

I think youre over complicating this @Draco3. If each of the various streams contains tags, you can use the route filters to process only the individual streams of data.

For example:

Route 1: type == 'windows'
Route 2: type == 'firewall'
Route 3: type == 'antivirus'
Route 4: true

Make sure to set each route with the Final flag to Yes, otherwise data will continue to the next route and not be stopped by the filter. Read more on the docs here: Routes | Cribl Docs

Each route will only process the respective type of data.