We have updated our Terms of Service, Code of Conduct, and Addendum.

Is there a limit to file size when importing data from object stores?

Jon Rust
Jon Rust Posts: 475 mod
in General Discussions

Are there any limitations to the file size or events when importing data using an object store collector (s3, azure, gcp, on-prem obj store)?

Best Answer

  • Jon Rust
    Jon Rust Posts: 475 mod
    Answer ✓

    Individual event size limitations are controlled by the event breaker definitions applied to the Source. Default is 50 KB max size. Overall files (objects) are not limited per se, but remember that objects have to be downloaded before events can be extracted. And while decompression is done in a streaming fashion so disk space wont be all that impacted, a large file can take more CPU cycles than expected.

Answers

  • Jon Rust
    Jon Rust Posts: 475 mod
    Answer ✓

    Individual event size limitations are controlled by the event breaker definitions applied to the Source. Default is 50 KB max size. Overall files (objects) are not limited per se, but remember that objects have to be downloaded before events can be extracted. And while decompression is done in a streaming fashion so disk space wont be all that impacted, a large file can take more CPU cycles than expected.

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    Max event size is predicated on the breaker used. The default breaker supports event size up to 51,200 bytes. If events are expected to be bigger than that the Cribl - Do not break ruleset can be used, which supports up to 1MB events, but if that doesnt properly break the events being replayed then a completely custom breaker will be required. Max event size for a customer breaker is 134,217,728 bytes.

    As for max file size, no limit exists based on the info from the Software engineering team.

  • Chris
    Chris Posts: 13 mod

    One thing that Ive noticed is that if you have a very large raw payload from an S3 collector (say like an event that is 20+MB), there seems to be some issue with the API worker process on the Leader being able to handle that.

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    Yeah Im aware of that issue that Jeremy is working on with you. Were still investigating that so well have more information once we determine the root cause of that. But at present there arent any Cribl defined limits on the files but there is for the event size.

Categories