Does Collect has a checkpoint option?

mtorres30

Hello! I am using HTTP Discover and Collect with Login Authentication, to take the notifications.log. Then I pass them to a route and send them to Splunk. Question: Does Collect has a checkpoint option? Currently, every run on collect, gets the entire file every time, so getting duplicates.

Tomer Shvueli

the REST Collector does not have a checkpoint option. when you say notifications.log, are you referring to an endpoint/API or a file? there are some options to not collect duplicates with either endpoint or file collection, such as <https://docs.cribl.io/stream/collectors-schedule-run#about-partitions-and-tokens|leveraging Partitions and Tokens> if you're able to separate files by date for example