We have updated our Terms of Service, Code of Conduct, and Addendum.

Split AZURE event Hub records into individual events

I have tried Event Breaker on this type of data but it didn't work, it can't separate the record into individual events

note: this is the live data streamed from Azure event hub

and this is the event breaker configuration

but if I use the same pipeline or the event breaker at the knowledge with the data in that shape it works fine and splits every record into an event

I tried to use unroll but also didn't work so any suggestion on how can i solve this issue?

thanks,

Comments

  • Jon Rust
    Jon Rust Posts: 476 mod

    I'm not following the problem statements:

    > I have tried Event Breaker on this type of data but it didn't work, it can't separate the record into individual events … but if I use the same pipeline or the event breaker at the knowledge with the data in that shape it works fine and splits every record into an event

    Can you clarify?

  • I have created a pipeline with the event breaker function if I applied it to the live data this what will happen to the data sample

    the Input data

    the output

    but if I copy past the live data sample into event breaker rules with the same configuration it will work fine

    Input :

    output

  • Jon Rust
    Jon Rust Posts: 476 mod

    Did you commit and deploy?

  • yes

  • Jon Rust
    Jon Rust Posts: 476 mod
    edited April 2024

    The EB function only operates on data in _raw. Your records[] field is not in _raw. You can use the Unroll function with Source and destination field configs listed as recordsor you can serialize recordsinto raw and then run the EB.

    https://docs.cribl.io/stream/event-breaker-function

    Using Unroll:


    With this starting data:

    Results in:

  • i did the same but it didn't work on the event with an inside record 659

    output

    i also attach a sample of the data

  • Jon Rust
    Jon Rust Posts: 476 mod

    With a destination field different from the original, you end up with 659 records with the original 659 records in each — meaning more than 400,000 records in all. This is likely too much data for the preview window, which runs in your browser's JS engine, to handle. Try using records as the destination for Unroll.

  • i get your point thank you so much it worked this was very helpful

  • yp
    yp Posts: 2

    I don't see how the EB plays in - I have configured the unroll and then if I add the EB next, the output did not have any difference comparing to the result of using unroll alone.

    I also don't have the cribl_breaker field shown up.

  • yp
    yp Posts: 2

    turns out I can if I change the output to "_raw".

    However, "JSON Extract Fields" toggled on and off have no difference in results.