The TCP output processor has paused the data flow

Thinus Swart

Afternoon (at least on my side).Need some help on my side. I've got a Splunk HF cluster sending events to my Cribl cluster. The HFs are experiencing this issue every now and then:
08-16-2023 09:29:32.056 +0200 WARN  TcpOutputProc [12084 indexerPipe_1] - The TCP output processor has paused the data flow. Forwarding to host_dest=xx.xx.xx.xx inside output group cribl_out_group from host_src=xxxxxx has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.It's not happening too often, so I guess the questions are:

1. Should I be worried?
2. If yes, where do I start looking for what might be causing the issue on the Cribl side?

Oliver Hoppe

Hi @Thinus Swart

Usually that happens when Cribl Stream is creating backpressure towards your HFs. Root cause in most cases is either that one of your downstream systems is set to block and is either down or cannot handle the load Cribl Stream is sending or your Stream instance have not sufficient compute.
Please check your Stream destinations health first

Oliver Hoppe

that depends

https://docs.cribl.io/stream/persistent-queues/#source-andor-destination-pq

if you have a single destination I would configure source PQ, if you have multiple use destination PQ as you can ensure that way that even if one destination is blocked your other destination(s) will get the data

Oliver Hoppe

Hi @Thinus Swart

Usually that happens when Cribl Stream is creating backpressure towards your HFs. Root cause in most cases is either that one of your downstream systems is set to block and is either down or cannot handle the load Cribl Stream is sending or your Stream instance have not sufficient compute.
Please check your Stream destinations health first

Thinus Swart

Okay, that makes sense. Would it make sense to enable PQ on my Splunk sources?

Oliver Hoppe

that depends

https://docs.cribl.io/stream/persistent-queues/#source-andor-destination-pq

if you have a single destination I would configure source PQ, if you have multiple use destination PQ as you can ensure that way that even if one destination is blocked your other destination(s) will get the data

Thinus Swart

Okay, sounds like destination PQ is the way to go then. Thanks