Cribl S3 Collector collect All the log files from S3 bucket everytime instead only new ones
I'm using cribl S3 collector to collect the logs from AWS S3 bucket. S3 bucket contains logs of Akamai datastream which has the log format as below.
ak- akamai file prefix
913478 - random string
170xxxxxxxx - EPOCH timestamp
008071 - random string
ds- file suffix
When I schedule the S3 collector to run every 15 minutes, it collects all the log files from buckets every time.
Looking for a suggestion, how do I collect only the new files from S3 not all the files every time.