We have updated our Terms of Service, Code of Conduct, and Addendum.

The TCP output processor has paused the data flow

Options
Thinus Swart
Thinus Swart Posts: 3
edited August 2023 in General Discussions

Afternoon (at least on my side).Need some help on my side. I've got a Splunk HF cluster sending events to my Cribl cluster. The HFs are experiencing this issue every now and then:
08-16-2023 09:29:32.056 +0200 WARN  TcpOutputProc [12084 indexerPipe_1] - The TCP output processor has paused the data flow. Forwarding to host_dest=xx.xx.xx.xx inside output group cribl_out_group from host_src=xxxxxx has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.It's not happening too often, so I guess the questions are:

  1. Should I be worried?
  2. If yes, where do I start looking for what might be causing the issue on the Cribl side?

Best Answers

  • Oliver Hoppe
    Oliver Hoppe Posts: 50 ✭✭
    Answer ✓
    Options

    Hi @Thinus Swart

    Usually that happens when Cribl Stream is creating backpressure towards your HFs. Root cause in most cases is either that one of your downstream systems is set to block and is either down or cannot handle the load Cribl Stream is sending or your Stream instance have not sufficient compute.
    Please check your Stream destinations health first

  • Oliver Hoppe
    Oliver Hoppe Posts: 50 ✭✭
    Answer ✓
    Options

    that depends

    https://docs.cribl.io/stream/persistent-queues/#source-andor-destination-pq

    if you have a single destination I would configure source PQ, if you have multiple use destination PQ as you can ensure that way that even if one destination is blocked your other destination(s) will get the data

Answers

  • Oliver Hoppe
    Oliver Hoppe Posts: 50 ✭✭
    Answer ✓
    Options

    Hi @Thinus Swart

    Usually that happens when Cribl Stream is creating backpressure towards your HFs. Root cause in most cases is either that one of your downstream systems is set to block and is either down or cannot handle the load Cribl Stream is sending or your Stream instance have not sufficient compute.
    Please check your Stream destinations health first

  • Thinus Swart
    Thinus Swart Posts: 3
    Options

    Okay, that makes sense. Would it make sense to enable PQ on my Splunk sources?

  • Oliver Hoppe
    Oliver Hoppe Posts: 50 ✭✭
    Answer ✓
    Options

    that depends

    https://docs.cribl.io/stream/persistent-queues/#source-andor-destination-pq

    if you have a single destination I would configure source PQ, if you have multiple use destination PQ as you can ensure that way that even if one destination is blocked your other destination(s) will get the data

  • Thinus Swart
    Thinus Swart Posts: 3
    Options

    Okay, sounds like destination PQ is the way to go then. Thanks