We have updated our Terms of Service, Code of Conduct, and Addendum.

Optimize sourcetype assignment in pipeline

We have a source coming in that consists of many different sourcetypes all under the same sourcetype, and we want to assign sourcetypes in a pipeline. E.g. an event starts with `2023-06-15T00:00` it's going to be sourcetype A, another one starts with `INFO 06/15/23` that's sourcetype B and so on. How would I best set up the functions in the pipeline, so that when one of the assignments was made the others aren't tried (for logical and performance reasons - an event can only ever have one sourcetype)? I guess we could make the function final, but then other functions (after the sourcetype has been assigned) aren't applied either, e.g. aggreations on some of these sourcetypes. Should I create two pipelines, one just for assigning the sourcetype with final functions, and then a follow up pipeline for further processing?


  • David Maislin
    David Maislin Posts: 228 mod

    Do that in the event breakers and just add sourcetype fields at the bottom of the ruleset.

  • Why do you recommend that? Unfortunately, some of the sourcetype assignments depend on parsing fields from json first

  • David Maislin
    David Maislin Posts: 228 mod

    Use Filters at the Route and a pipeline for each type of event?

  • dritan
    dritan Posts: 51 mod

    can do that by checking for lack of sourcetype. i.e., if the event has a sourcetype that processing (in that Function) will be skipped

  • The events come in with a sourcetype already, but yeah good idea to check existence of more specific sourcetype!