We have updated our Terms of Service, Code of Conduct, and Addendum.

Technical question on stream file monitor

SonOfBuzi
SonOfBuzi Posts: 12 ✭
edited March 2023 in General Discussions

i have a use case where we have a nfs share that applications writes logs too i would like to monitor these while being written to (file monitor?) but distribute the load across workers. is that possible?

Tagged:

Best Answer

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod
    Answer ✓

    Depending on the folder structure/file naming pattern of the data in the NFS mount, you could use tokens to only look for new events in certain folders. Collection will only forward events inside the time range specified once they are properly broken with the configured event breaker.

Answers

  • Jon Rust
    Jon Rust Posts: 475 mod

    Not really. Each worker in a group* gets the same configs and will try to read the same files. There is no communication between them. The File System collector is a little different, but its not going to keep track of what its already seen.

    My rec would be to install Edge or Stream on a single instance worker group and have it responsible for reading the files on that share.

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod
    Answer ✓

    Depending on the folder structure/file naming pattern of the data in the NFS mount, you could use tokens to only look for new events in certain folders. Collection will only forward events inside the time range specified once they are properly broken with the configured event breaker.

Categories