We have updated our Terms of Service, Code of Conduct, and Addendum.

How do you get the workers to send metrics (like CPU usage, etc.) to Splunk?

Jon Rust
Jon Rust Posts: 443 mod
in General Discussions

Noob question here... How do you get the workers to send metrics (like CPU usage, etc.) to Splunk? We have the Leader sending cribl metrics, but not the workers.

Answers

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    Enable the internal Cribl Metrics input to send them through the pipelines.

  • Jon Rust
    Jon Rust Posts: 443 mod

    Since I didn't set up the Cribl env, I will have to look to see how that leader was set up, I guess to see how to do that for the workers.

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    Not sure what you mean by the leader sending metrics. The leader doesn't send out any metrics.

  • Jon Rust
    Jon Rust Posts: 443 mod

    0

  • Jon Rust
    Jon Rust Posts: 443 mod

    These are going to the cribl_logs index. The cpuPerc is what I'm most interested in.

  • Jon Rust
    Jon Rust Posts: 443 mod

    I think....

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    Those are only logs. Not actual metrics in a metrics format. Metrics need to go to a metrics index. If what you have there is sufficient then you won't need the cribl metrics source enabled but you'll want the cribl logs source enabled instead.

  • Jon Rust
    Jon Rust Posts: 443 mod

    So, this is what we have that i would consider the metrics going to a metrics index, but there is nothing in the index (which is a metrics index):

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    Is the cribl metrics source enabled ?

  • Jon Rust
    Jon Rust Posts: 443 mod

    I didn't set that up, BTW, so please don't think I know why it was set up this way. :wink:

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    Go into the inputs for each worker group to see if that input is on.

  • Jon Rust
    Jon Rust Posts: 443 mod

    How do I find that out?

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    yes

  • Jon Rust
    Jon Rust Posts: 443 mod

    From the Sources -> Cribl Internal -> CriblMetrics, they are enabled. Is that what you mean?

  • Jon Rust
    Jon Rust Posts: 443 mod

    So if they are already enabled, why would I not see the metrics in that index?

  • Jon Rust
    Jon Rust Posts: 443 mod

    Thank you for your help, BTW. I truly appreciate it.

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    hard to say, i've only seen a small section of your configuration. Maybe a route for them isn't setup? Maybe they aren't even going thru that pipeline.

  • Jon Rust
    Jon Rust Posts: 443 mod

    Route is there.

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    I recommend filing a support case so we can review the entire config in depth and possibly conduct a zoom call if necessary. Please include a diag bundle from a worker node (not the leader node) for our review.

  • Jon Rust
    Jon Rust Posts: 443 mod

    Pipeline:

  • Jon Rust
    Jon Rust Posts: 443 mod

    Gov't won't let out diags that aren't completely cleaned and then they have to go through a clearance process. But I can file a support case, and do a screen share.

Categories