We have updated our Terms of Service, Code of Conduct, and Addendum.

Are there default limits for JSON processing and are they tunable?

Jeremy Prescott
Jeremy Prescott Posts: 33 ✭✭
edited September 2023 in General Discussions

I have a TCP JSON stream with massively long lines… i suspect Cribl is truncating them. Are there default limits for JSON processing? and are they tunable?

Tagged:

Answers

  • Jeremy Prescott
    Jeremy Prescott Posts: 33 ✭✭

    i did not. what event breaker might it be using since I didn't define one?

  • David Maislin
    David Maislin Posts: 228 mod

    Did you create an event breaker and adjust the max event length?

  • David Maislin
    David Maislin Posts: 228 mod
    edited September 2023

    You can go under Knowledge/Event Breakers and create a new RuleSet

    https://cribl.io/blog/mastering-event-breaking-management-with-cribl/

  • Jeremy Prescott
    Jeremy Prescott Posts: 33 ✭✭

    from that it appears "JSON Newline Delimited" is the default, and would apply in my case

  • Jeremy Prescott
    Jeremy Prescott Posts: 33 ✭✭

    i'm sending via TCPJSON, in which each event is newline delimited (not pretty-printed). that appears to be working

  • David Maislin
    David Maislin Posts: 228 mod

    Have you considered using the TCP source instead which supports EventBreakers?

  • Jeremy Prescott
    Jeremy Prescott Posts: 33 ✭✭
    edited September 2023

    "max event bytes " appears to be 1024000 by default in the "JSON Newline Delimited" breaker.. so that must be what i'm up against

    so the TCPJSON source does not support event breakers? meaning the default "cribl" event breaker (with ndjson rule) is not even being applied?

  • David Maislin
    David Maislin Posts: 228 mod
    edited September 2023

    we just using fallback breaker on new lines on TCP JSON right?

    I haven't tried, but have you tried upping the ndson ruleset to: `134217728`?

    If not, feel free to test with the TCP source and do an event breaker there.

  • Jeremy Prescott
    Jeremy Prescott Posts: 33 ✭✭

    i haven't touched it since I'm just now learning about these limits. what is that number? seems very specific

  • David Maislin
    David Maislin Posts: 228 mod
    edited September 2023

    It is the max size of an event we support in bytes.

    128 MB

  • Jeremy Prescott
    Jeremy Prescott Posts: 33 ✭✭
    edited September 2023

    oohhh. ok well then that's a hard limit across all sources / destinations?

    that's very large but I'll have to see if any single event we have exceeds that

  • David Maislin
    David Maislin Posts: 228 mod

    That is the biggest event size we currently support IIRC.

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod
    edited September 2023

    to answer your question, probably (I'm assuming it's). It doesn't look like TCP JSON source can have an event breaker configured for it so he'll probably have to switch over to the raw TCP to use an event breaker with a larger max event size even though the event delimiter () would be the same.

  • David Maislin
    David Maislin Posts: 228 mod
    edited September 2023

    Thanks

  • Jeremy Prescott
    Jeremy Prescott Posts: 33 ✭✭

    That's interesting. Any particular reason there's not configurable event breaker for TCP JSON sources?

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    not that i'm aware

  • Jeremy Prescott
    Jeremy Prescott Posts: 33 ✭✭
    edited September 2023

    alright cool, thanks guys. I'll test but assuming nothing else pops up, sounds like it should work

    i've implemented this but now it's deleted my _raw field in every event

  • David Maislin
    David Maislin Posts: 228 mod

    That's very odd. Maybe need to tweak some things. I can Zoom later if you like.

  • Jeremy Prescott
    Jeremy Prescott Posts: 33 ✭✭
    edited September 2023

    nevermind… i was putting __raw into _raw via TCPJSON, but __raw doesn't exist in TCP sources

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    Yep indeed. Only _raw exists.

Categories