A few question about PQ
A few question about PQ: Customer is performing a data center move. There backend will be unavailable for a few hours. Therefore, the customer wants to utilize PQ in Cribl. He needs to enable PQ first. Is there an option to globally enable or disable PQ? (There doesn't seem to be a button, but perhaps a simple API solution?) Is there an option to enable or disable PQ for all sources under the same "technology"? For example, all Syslog sources. What is better, source PQ or dest PQ? For destination PQ it is set once (he has only one destination for all sources), will it have an impact when destination comes back online? Would Source PQ be better for Cribl? Should both be set? What happens when PQ will be disabled? Will the data still in PQ be sent, or will it stay there forever?
Best Answer
-
You should be just fine without it.
0
Answers
-
For down destination, use the Destination PQ. How many workers are there? No option for global PQ, but since there is a single destination, you can enable that there. Use compression and ensure your workers have enough disk space to account for the total amount of data you receive over a few hours, then double or triple that space just to be sure. Compression buys you about 8X the volume. Once the destination is back online and the workers can connect the queue will empty. Yes there is a bit of a performance hit across the workers. You could always add a couple of workers to scale horizontally and after all the queues are empty, make them go away. When you disable PQ I am reasonably certain it will empty the queue.
0 -
about 10 workers, 2-3 per worker group. worker groups are set for different network segments. there should be plenty of pq disk space. the customer prepares a terrabyte of nfs disk space. the customer has to "give back" the disk space sooner or later. thats why I asked, what happens if pq is disabled if the backend is available again. At some point the customer want to disable pq to make sure no data is written to the disk but also want to make sure, no old data is still on the pq storage
0 -
By default, after data flow is re-established, Cribl Stream will forward events in FIFO (first in, first out) order: It will send out earlier queued events before newly arriving events. In Cribl Stream 4.1 and later, you can instead prioritize new events by disabling Destinations' Strict ordering control.Checking on just turning off PQ behavior now
0 -
ok
0 -
If you just were to leave it on and the Destination comes back on line, it will empty completely and not fill with additional events.
But still checking on turning it off, what happens.
Looks like leaving it on until the queue is empty is the answer. The events stay persisted to disk but are not sent out unless you enable PQ again0 -
I'm more afraid that backpressure could happen from time to time and that the PQ is then never cleared because PQ writes new events to the disk
0 -
Backpressure is not a down destination. You could also enable source PQ to deal with that.
0 -
yes but then i've to edit 40-50 sources
0 -
and enable/disable …
0 -
You should be just fine without it.
0