We have updated our Terms of Service, Code of Conduct, and Addendum.

Rest API - Logs not going to destination. Preview/Discover work.

Options
michael.daniels@hannaandersson.com
michael.daniels@hannaandersson.com Posts: 3 ✭
in Stream

I'm using a Rest API to grab logs from a cloud provider. I can see the logs are pulled from the API. I can see that the data is formatted correctly.

Within my Data Source.

Result Routing → Sent to Routes: NO Pipeline: passthru Destination: SIEM

Preview [Before Destination]: Shows my data formatted correctly.

Discover: Sends a few lines to my SIEM

Full Run: Nothing happens.

Any ideas on what to look at to figure this out or how I can troubleshoot this would be greatly appreciated.

Thanks!

Comments

  • Jon Rust
    Jon Rust Posts: 439 mod
    edited March 1
    Options

    Try disabling time filtering (under advanced) and see if that helps. If it does, configure an Event Breaker and configure time extraction from the payload.

    Edit: fixed bad editing/typos

  • michael.daniels@hannaandersson.com
    michael.daniels@hannaandersson.com Posts: 3 ✭
    Options

    Thanks Jon!

    I ended up tracing the source, pipeline and destination. It looks like there was too much data from the API to send to the endpoint.

    I adjusted the time interval when I scheduled it to just pick up the last 5 minutes, and scheduled the job every five minutes.

    Now I'm getting the logs we need and its all in the correct format. The long API returns were causing all sorts of issues.

    Your Videos on Youtube helped too! Appreciate all your help!

Categories