Events in flight (and memory queues/buffers)
An interesting topic came up yesterday which I didn’t have a good answer too, around how many events are in flight in Stream at any point in time.
It was a discussion around potential for data loss, and if something adverse were to happen the worker, what the impact would be. I imagine there is up to one event being processed per worker thread at any point in time, but is there a buffer of events before the worker thread picks it up? If so, how much, and if not, does the thread just block the network socket until it’s done processing?
Supplementary to that - what happens when a destination goes down? Persistent queues aside, how much will Cribl buffer before applying back-pressure to the pipeline?
And probably scenarios I haven’t considered. How much data is it theoretically possible to have in flight at any point in time, and thus risk of loss in case of catastrophe?