We have updated our Terms of Service, Code of Conduct, and Addendum.

Using Persistent Queue drops thruput

Options
Mike Dupuis
Mike Dupuis Posts: 14 admin
in General Discussions

Hi All, I am having some issues when enabling a Smart Persistent Queue for a Splunk TCP source (TLS disabled). When the persistent queue is enabled the thruput drops by 2 orders of magnitude (from ~20k eps to ~200eps). The Cribl worker nodes are running on VMs, could this be due to the storage not being fast enough? If this is the reason why would the thruput drop so dramatically? Does enabling persistent queues on a source use much CPU?I don’t have any other source persistent queues enabled currently, but do have some on destinations that are working fine.

Tagged:

Best Answer

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod
    Answer ✓
    Options

    PQ required the cpu to be involved in order for the application to interact with the hardware. So you’ll be taking a hit it read and write with PQ. That steals cycles away from the cpu that would otherwise be processing events.
    Storage can definitely affect the throughput.I recommend using Always On mode for source PQ. Depending on your config your smart mode may be blocking a lot and Always On won’t do that unless the storage is simply full.  We also have some performance issues with Smart mode so Always On is recommended for now.

Answers

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod
    Answer ✓
    Options

    PQ required the cpu to be involved in order for the application to interact with the hardware. So you’ll be taking a hit it read and write with PQ. That steals cycles away from the cpu that would otherwise be processing events.
    Storage can definitely affect the throughput.I recommend using Always On mode for source PQ. Depending on your config your smart mode may be blocking a lot and Always On won’t do that unless the storage is simply full.  We also have some performance issues with Smart mode so Always On is recommended for now.

Categories