We want to enable persistent queuing and want to know how large we can make the max file size?
We are enabling persistent queuing (PQ) on our destinations and notices that the Max File size is defaulted to 1MB. With the amount of data we are processing that will lead to A LOT of small files so we were thinking of increasing that to a Max File Size of 100MB. However, before we do that - we wanted to know if there was any potential impact to Cribl if the PQ files are to large ?
When Cribl is reading these files to send to the destination, does it read the entire file into memory ?