We have updated our Terms of Service, Code of Conduct, and Addendum.

How do you get the workers to send metrics (like CPU usage, etc.) to Splunk?

Noob question here... How do you get the workers to send metrics (like CPU usage, etc.) to Splunk? We have the Leader sending cribl metrics, but not the workers.

Answers

  • Enable the internal Cribl Metrics input to send them through the pipelines.

  • Jon Rust
    Jon Rust Posts: 426 mod

    Since I didn't set up the Cribl env, I will have to look to see how that leader was set up, I guess to see how to do that for the workers.

  • Not sure what you mean by the leader sending metrics. The leader doesn't send out any metrics.

  • Jon Rust
    Jon Rust Posts: 426 mod

    0

  • Jon Rust
    Jon Rust Posts: 426 mod

    These are going to the cribl_logs index. The cpuPerc is what I'm most interested in.

  • Jon Rust
    Jon Rust Posts: 426 mod

    I think....

  • Those are only logs. Not actual metrics in a metrics format. Metrics need to go to a metrics index. If what you have there is sufficient then you won't need the cribl metrics source enabled but you'll want the cribl logs source enabled instead.

  • Jon Rust
    Jon Rust Posts: 426 mod

    So, this is what we have that i would consider the metrics going to a metrics index, but there is nothing in the index (which is a metrics index):

  • Jon Rust
    Jon Rust Posts: 426 mod

    I didn't set that up, BTW, so please don't think I know why it was set up this way. :wink:

  • Is the cribl metrics source enabled ?

  • Jon Rust
    Jon Rust Posts: 426 mod

    How do I find that out?

  • Go into the inputs for each worker group to see if that input is on.

  • Jon Rust
    Jon Rust Posts: 426 mod

    From the Sources -> Cribl Internal -> CriblMetrics, they are enabled. Is that what you mean?

  • yes

  • Jon Rust
    Jon Rust Posts: 426 mod

    So if they are already enabled, why would I not see the metrics in that index?

  • Jon Rust
    Jon Rust Posts: 426 mod

    Thank you for your help, BTW. I truly appreciate it.

  • hard to say, i've only seen a small section of your configuration. Maybe a route for them isn't setup? Maybe they aren't even going thru that pipeline.

  • Jon Rust
    Jon Rust Posts: 426 mod

    Route is there.

  • Jon Rust
    Jon Rust Posts: 426 mod

    Pipeline:

  • I recommend filing a support case so we can review the entire config in depth and possibly conduct a zoom call if necessary. Please include a diag bundle from a worker node (not the leader node) for our review.

  • Jon Rust
    Jon Rust Posts: 426 mod

    Gov't won't let out diags that aren't completely cleaned and then they have to go through a clearance process. But I can file a support case, and do a screen share.