Logstream Filesystem output
The Filesystem destination allows setting a prefix and suffix for the filename. However, I don’t see a place to set the actual base filename, or to just not have one. I don’t want the 6 random character base filename. How can this be removed? Also, I’d prefer not to have file staging, since I am not compressing files, is there a way to just write raw output to disk in real-time? Using the staging dir creates latency.
Best Answer
-
Randomness is required in filenames destination to avoid race conditions as there are multiple worker processes writing data to files. As Dritan mentions, it is also required to avoid conflicts when moving to the final directory. The filesystem output is optimized for speed the extra logic required to move vs. append to the output file would have a negative impact on performance.
The staging directory is required because the filesystem destination is required to buffer many megabytes of data per file and worker process before moving to the final destination. Think of the staging directory as a filesystem backed buffer. If we remove the staging directory this destinations memory footprint would be greatly increased as the events would need to be buffered in memory.
0
Answers
-
IIRC the random 6 chars in the middle of the filename are needed by the application to ensure moving to the final directory happens without conflicts.
Staging directories are required for non-streaming destinations like Filesystem, S3 etc. Whats the requirement for latency in this case? You can tweak Advanced Settings or or if the consuming application can receive events in real-time I suggest you consider another type of Destination such as TCPJSON etc.0 -
Randomness is required in filenames destination to avoid race conditions as there are multiple worker processes writing data to files. As Dritan mentions, it is also required to avoid conflicts when moving to the final directory. The filesystem output is optimized for speed the extra logic required to move vs. append to the output file would have a negative impact on performance.
The staging directory is required because the filesystem destination is required to buffer many megabytes of data per file and worker process before moving to the final destination. Think of the staging directory as a filesystem backed buffer. If we remove the staging directory this destinations memory footprint would be greatly increased as the events would need to be buffered in memory.
0
