Split AZURE event Hub records into individual events
I have tried Event Breaker on this type of data but it didn't work, it can't separate the record into individual events
note: this is the live data streamed from Azure event hub
and this is the event breaker configuration
but if I use the same pipeline or the event breaker at the knowledge with the data in that shape it works fine and splits every record into an event
I tried to use unroll but also didn't work so any suggestion on how can i solve this issue?
thanks,
Comments
-
I'm not following the problem statements:
> I have tried Event Breaker on this type of data but it didn't work, it can't separate the record into individual events … but if I use the same pipeline or the event breaker at the knowledge with the data in that shape it works fine and splits every record into an eventCan you clarify?
0 -
I have created a pipeline with the event breaker function if I applied it to the live data this what will happen to the data sample
the Input data
the output
but if I copy past the live data sample into event breaker rules with the same configuration it will work fine
Input :
output
0 -
Did you commit and deploy?
0 -
yes
0 -
The EB function only operates on data in _raw. Your
records[]
field is not in _raw. You can use the Unroll function with Source and destination field configs listed asrecords
or you can serializerecords
into raw and then run the EB.Using Unroll:
With this starting data:Results in:
0 -
i did the same but it didn't work on the event with an inside record 659
output
i also attach a sample of the data
0 -
With a destination field different from the original, you end up with 659 records with the original 659 records in each — meaning more than 400,000 records in all. This is likely too much data for the preview window, which runs in your browser's JS engine, to handle. Try using records as the destination for Unroll.
1 -
i get your point thank you so much it worked this was very helpful
1 -
I don't see how the EB plays in - I have configured the unroll and then if I add the EB next, the output did not have any difference comparing to the result of using unroll alone.
I also don't have the cribl_breaker field shown up.
0 -
turns out I can if I change the output to "_raw".
However, "JSON Extract Fields" toggled on and off have no difference in results.0