We have updated our Terms of Service, Code of Conduct, and Addendum.

Split AZURE event Hub records into individual events

Options
nada28.7.1996@gmail.com
nada28.7.1996@gmail.com Posts: 5 ✭
in Stream

I have tried Event Breaker on this type of data but it didn't work, it can't separate the record into individual events

note: this is the live data streamed from Azure event hub

image.png

and this is the event breaker configuration

image.png

but if I use the same pipeline or the event breaker at the knowledge with the data in that shape it works fine and splits every record into an event

image.png

I tried to use unroll but also didn't work so any suggestion on how can i solve this issue?

thanks,

Comments

  • Jon Rust
    Jon Rust Posts: 439 mod
    Options

    I'm not following the problem statements:

    > I have tried Event Breaker on this type of data but it didn't work, it can't separate the record into individual events … but if I use the same pipeline or the event breaker at the knowledge with the data in that shape it works fine and splits every record into an event

    Can you clarify?

  • nada28.7.1996@gmail.com
    nada28.7.1996@gmail.com Posts: 5 ✭
    Options

    I have created a pipeline with the event breaker function if I applied it to the live data this what will happen to the data sample

    the Input data

    image.png

    the output

    image.png

    but if I copy past the live data sample into event breaker rules with the same configuration it will work fine

    Input :

    image.png

    output

    image.png

  • Jon Rust
    Jon Rust Posts: 439 mod
    Options

    Did you commit and deploy?

  • nada28.7.1996@gmail.com
    nada28.7.1996@gmail.com Posts: 5 ✭
    Options

    yes

  • Jon Rust
    Jon Rust Posts: 439 mod
    edited April 22
    Options

    The EB function only operates on data in _raw. Your records[] field is not in _raw. You can use the Unroll function with Source and destination field configs listed as recordsor you can serialize recordsinto raw and then run the EB.

    https://docs.cribl.io/stream/event-breaker-function

    Using Unroll:

    image.png


    With this starting data:

    image.png

    Results in:

    image.png

  • nada28.7.1996@gmail.com
    nada28.7.1996@gmail.com Posts: 5 ✭
    Options

    i did the same but it didn't work on the event with an inside record 659

    image.png

    output

    image.png
    AzureEventHub_Record659.txt

    i also attach a sample of the data

  • Jon Rust
    Jon Rust Posts: 439 mod
    Options

    With a destination field different from the original, you end up with 659 records with the original 659 records in each — meaning more than 400,000 records in all. This is likely too much data for the preview window, which runs in your browser's JS engine, to handle. Try using records as the destination for Unroll.

  • nada28.7.1996@gmail.com
    nada28.7.1996@gmail.com Posts: 5 ✭
    Options

    i get your point thank you so much it worked this was very helpful

Categories