We have updated our Terms of Service, Code of Conduct, and Addendum.

Is anyone sending PerfmonMetrics through Cribl?

2»

Answers

  • Tony Reinke - Cribl
    Tony Reinke - Cribl Posts: 134 admin

    <@U03FQSY3JCF&gt; i had similar issue with my udp & perfmon metrics data using passthru to Splunk Cloud after upgrading to Cribl Stream 4.0.0 and updating our Splunk Cloud destination's max s2s version to use v4 instead of v3. My currently working solution to continue getting my metric data through Cribl Stream into our Splunk Cloud indexers was to clone my Splunk Cloud destination to another one and set s2s to use v3 instead of v4, filter these events out with a new Route using passthru pipeline and output that route to the cloned v3 Splunk Cloud destination. We've submitted a support case for this, i've recreated this by also sending the same metrics using passthru with s2s v3 to a Splunk Enterprise dev instance and also fails to get there when i flip s2s to v4

  • morrisnky
    morrisnky Posts: 16 ✭

    But do not use the "hack" - things blow up and it messes with Splunk processing queues and you can bjork the metrics components, so you cannot even troubleshoot or fix. Use `raw` HEC endpoint, if you want Splunk to do some index-time parsing. Personally, I try best to use push-based with Splunk as a destination with HEC over SplunkTCP. Better on your network and the perf has dramtically improved from early days.

  • jlstanley
    jlstanley Posts: 21 ✭

    would one of you mind sending me what the HEC metrics format for PerfmonMetrics should look like to properly send to splunk cloud's hec endpoint? I'm trying this but it's not working: ```{ "source": "Perfmon:Network", "host": "W2022GOLDTEST5", "sourcetype": "PerfmonMetrics:Network", "index": "win_em_metrics", "entity_type": "Windows_Host", "time": 1674495318, "event": "metric", "fields": { "_value": "23888.970133604955", "metric_name": "Network Interface.Bytes Total/sec", "collection": "Network", "instance": "vmxnet3 Ethernet Adapter" }, "cribl_pipe": "perfmon_metrics" }```

  • morrisnky
    morrisnky Posts: 16 ✭

    Are you using the <https://docs.cribl.io/stream/publish-metrics-function/|Publish Metrics> function to ensure all you dims and values are correct, before sending to HEC destination?

  • jlstanley
    jlstanley Posts: 21 ✭

    No, i thought if I created the fields that needed to be in the event and removed _raw then it would already be in the right format. is that not accurate? I haven't tried the Publish metrics function yet but can give it shot.

  • morrisnky
    morrisnky Posts: 16 ✭

    That is the most sure fire way I have got Splunk to play nice with metrics. Takes some tinkering to ensure you get the dims etc in right format.

  • morrisnky
    morrisnky Posts: 16 ✭

    Not that I can see, but you want it to look similar to the below example, before it leaves Stream.

  • jlstanley
    jlstanley Posts: 21 ✭

    ok I'll take a look and see if I can make some progress. are there any glaring issues you can see on the format I pasted above from a field perspective?

  • jlstanley
    jlstanley Posts: 21 ✭

    isn't that format for log to metrics vs what perfmon metrics should look like?

  • jlstanley
    jlstanley Posts: 21 ✭

    any chance you can paste your json of the pipeline you use for your perfmon metrics from cribl?

  • morrisnky
    morrisnky Posts: 16 ✭

    Nope, that is what the Publish Metrics function will give you, and ensure it is sent as a metrics, rather than event.

  • morrisnky
    morrisnky Posts: 16 ✭

    The ones I have done are locked in a secure customer env - sorry. If you can give me some redacted ones - I could try and replicate.

  • morrisnky
    morrisnky Posts: 16 ✭

    <@U03FQSY3JCF&gt; - check this thread out - https://cribl-community.slack.com/archives/CPYBPK65V/p1653030903605899

Categories