Optimize sourcetype assignment in pipeline
We have a source coming in that consists of many different sourcetypes all under the same sourcetype, and we want to assign sourcetypes in a pipeline. E.g. an event starts with `2023-06-15T00:00` it's going to be sourcetype A, another one starts with `INFO 06/15/23` that's sourcetype B and so on. How would I best set up the functions in the pipeline, so that when one of the assignments was made the others aren't tried (for logical and performance reasons - an event can only ever have one sourcetype)? I guess we could make the function final, but then other functions (after the sourcetype has been assigned) aren't applied either, e.g. aggreations on some of these sourcetypes. Should I create two pipelines, one just for assigning the sourcetype with final functions, and then a follow up pipeline for further processing?
Answers
-
Do that in the event breakers and just add sourcetype fields at the bottom of the ruleset.
0 -
Why do you recommend that? Unfortunately, some of the sourcetype assignments depend on parsing fields from json first
0 -
Use Filters at the Route and a pipeline for each type of event?
0 -
can do that by checking for lack of sourcetype. i.e., if the event has a sourcetype that processing (in that Function) will be skipped
0 -
The events come in with a sourcetype already, but yeah good idea to check existence of more specific sourcetype!
0
