We have updated our Terms of Service, Code of Conduct, and Addendum.

How to make a variable from one event and add that variable to all events in the pipeline?

Hello, is there a way to make a variable from one event in a pipeline and add that variable to all events in the pipeline? So if one event has "Name=John Smith" but no other event from that file has the Name field, can I add "Name=John Smith" to all events? I have information at the beginning of a file that has fields I want added to all events, and after that "header" information are the actual events. So if this is something that can be done at the event breaker level then great.

Answers

  • Jon Rust
    Jon Rust Posts: 458 mod

    you'd need to use a redis function for this i think.

  • Jon Rust
    Jon Rust Posts: 458 mod

    which means you'll need a redis server. And cluster, to be honest.

  • davec
    davec Posts: 6 mod

    If these events are all coming in together (for example in an JSON that has an array records it contains), you could do the event breaking in a pipeline function (or maybe unroll). But basically you would need the set of events to be together.

  • I originally read in the whole file as an event, then extracted the beginning variables, then parsed the data from there. This worked, but not for files that get bigger than the event size limit.

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod

    Default event size might be kind of low for your needs. Max configurable is ~134 MB/event. Are your events larger than that?

  • Looking at the files again, I don't think they're bigger than that. I'll test out an event breaker with the max configurable event size