Crowdstream - Bytes out size doubled

kiran k · January 31

Hi Everyone

We have integrated Azure Storage Account and Azure Application Gateway with CrowdStream (Cribl Stream) via EventHub. Since logs are in nested JSON format, we used unroll function to convert them into individual events before forwarding them to CrowdStrike NGSIEM.

Currently, we are observing that Bytes Out size is twice that of Bytes In. While we understand that event count should double, our concern is why Bytes Out size is also doubling compared to Bytes In. Additionally, we have not observed any duplicate events in CrowdStrike NGSIEM.

Jon Rust · January 31

A sample of the original event, and the resulting event(s) would help a lot. My best guess without any samples is that your unroll is keep bits of the original event in each and every resulting "unrolled" event.

kiran k · February 3

Original.log.txt

Result logs.txt

Hi Jon,

Thanks for the response, As we couldnt capture sample input and output log. attached the random log files

Jon Rust · February 3

Can you share the pipeline(s)?

kiran k · February 3

created 1 Pipeline with function unroll and attached it to source pre-processing.

Please find attached screenshot for reference.

Jon Rust · February 3

I'm not clear why you're seeing this behavior, but try adding an Eval function before the Unroll. In the Eval, drop the __raw field. That's a double underscore raw. Let me know if that helps.

Edit: I've confirmed internally that __raw is included in the internal metrics and volume accounting. Use the Eval mentioned above to remove it to avoid this mistake. I'll raise a ticket to for the product team to look into this.

kiran k · February 5

HI Jon,

We did add eval first then unroll, Now i dont see —raw field in output log, However still I observe increase in Bytes Out size. Attached relevant screenshot for reference.

eval.png

Jon Rust · February 5

Do you have multiple outputs? If you're sending data to more than one destination, your output will be appropriately higher. What does the preview screen's inspection show you? (The bar chart icon at the top of the preview pane)

kiran k · February 6

No we dont have multiple output, one source has one destination. Also in inspection preview no much different in events IN and events OUT. Attached screenshot for reference.

Statistics.png

Pipeline_Diagnostics.png

Source&Destination.png