We have updated our Terms of Service, Code of Conduct, and Addendum.

Scheduled Collector discovers events, but does not collect

Jordan Perks
Jordan Perks Posts: 11 ✭
in Stream

I have a REST collector discovering events but not collecting them when scheduled. It works Ad-hoc, though.

Any idea what could be causing this? I do not see errors in the logs.

Best Answer

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod
    edited July 2023 Answer ✓

    Hey @Jordan, some REST APIs have built-in earliest and latest query string parameters or body content. When you specify the earliest and latest settings in a schedule for a collector, were redundantly checking the timestamps of each event after the events are run through the event breaker.

    Heres an example of a Cribl REST collector parameters configuration to get data from the Okta System Log endpoint. You can see in the Okta Developer docs that you can pass an earliest and latest timestamp to bound the events collected from the API. System Log | Okta Developer

    d4b8bbcbf5e11661744e8193d5a9a1c6f0eb898e.png

    For example, if I configure my schedule to run every 5 minutes, Im sending the since and until parameters in the URL, but Cribl is also checking the timestamp of each event to see if it is in the range. Weve seen some customers have issues with events being mistakenly discarded even though they have valid timestamps.

    57fbea226d65fa4e51ab7279834d1bc2ce166dfc.png

    What I would recommend is enabling the "Disable Time Filter" under advanced settings. This will ensure all discovered events are collected and we dont perform a redundant timestamp check when its already configured in the collect parameters.

    975c1b2836f3f573948cd785b40b2ade507223e3.png

Answers

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod
    edited July 2023 Answer ✓

    Hey @Jordan, some REST APIs have built-in earliest and latest query string parameters or body content. When you specify the earliest and latest settings in a schedule for a collector, were redundantly checking the timestamps of each event after the events are run through the event breaker.

    Heres an example of a Cribl REST collector parameters configuration to get data from the Okta System Log endpoint. You can see in the Okta Developer docs that you can pass an earliest and latest timestamp to bound the events collected from the API. System Log | Okta Developer

    d4b8bbcbf5e11661744e8193d5a9a1c6f0eb898e.png

    For example, if I configure my schedule to run every 5 minutes, Im sending the since and until parameters in the URL, but Cribl is also checking the timestamp of each event to see if it is in the range. Weve seen some customers have issues with events being mistakenly discarded even though they have valid timestamps.

    57fbea226d65fa4e51ab7279834d1bc2ce166dfc.png

    What I would recommend is enabling the "Disable Time Filter" under advanced settings. This will ensure all discovered events are collected and we dont perform a redundant timestamp check when its already configured in the collect parameters.

    975c1b2836f3f573948cd785b40b2ade507223e3.png

Categories