We have updated our Terms of Service, Code of Conduct, and Addendum.

Does anyone have experience using cribl to send prometheus metrics?

Does anyone have experience using cribl to send prometheus metrics? In the OTEL destination doc, it describes that events must conform to otel protobuf spec, but doesn't describe how this would happen exactly: https://docs.cribl.io/stream/destinations-otel/#protocol-and-transport-support Is there any automatic translation that can be applied, either on the source, some pipeline function, or a destination?

«1

Answers

  • David Maislin
    David Maislin Posts: 229 mod

    So you want to take unstructured data and covert it to an OTEL compatible format as metrics?

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    In this case, the metrics I'm receiving are generated by prometheus node exporter, so they should be nicely convertible

  • This can be done and I am about to implment this exact scenario(stream source `prometheus scraper` beautifully scrapes node exporter metrics and then further you can send it wherever(we are about to send it to a splunk metrics index). The reason we are doing this is just for 2 nodes is, these are facing the big bad internet and my infosec team insists that I can't get an agent like telegraf to push the data out to cribl stream(which is in a relatively safer network). Is there a specific reason why you are planning such an approach?

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    In my case I have some resource/security constrained hosts where I wouldn't want to install an otel collecter directly, but instead want to use cribl as a replacement since it's distributed, has queueing, etc.

  • Clint Sharp
    Clint Sharp Posts: 27 mod

    We have looked into this before. It's hard fwiw. Otel schema is very complex.

  • Clint Sharp
    Clint Sharp Posts: 27 mod

    <@U01J91JRGL9&gt; we had talked about documenting the schema that would be required in order to get that out? Do we have a JSON Schema that could be used for validation?

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    I know only so much about protobuf messaging, but it seems like the otel metrics.proto is reasonably straightforward: https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentelemetry/proto/metrics/v1/metrics.proto Does cribl already implement this? If not, is there a way I can install protobufjs and write a script function in a pipeline to do the encoding myself?

  • Clint Sharp
    Clint Sharp Posts: 27 mod

    Yes, we implement this. There are two protocols for Otel, one that uses JSON over HTTP and one that uses Protobuf. I believe we support both now.

  • Clint Sharp
    Clint Sharp Posts: 27 mod

    If you match the schema it should just flow.

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    I was looking for something in the OTEL destination config that indicated the encoding it would use, but then the docs mentioned it had to be already properly encoded before sending to the destination.

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    My understanding is that if I were using the OTEL collector, I would have other options for where to send my now-protobuf-formatted data as well. So in my case, I'm actually hoping to format the metrics in protobuf as though they were produced by an OTEL collector, but actually send the data to Kafka

  • Clint Sharp
    Clint Sharp Posts: 27 mod

    It uses Protobuf by default, but that's an implementation detail. Here are some JSON Schemas that can be used for validation (I think): https://github.com/open-telemetry/opentelemetry-configuration/tree/main/schema

  • nicktank
    nicktank Posts: 26 mod

    HTTP is overwhelmingly the protocol we see being adopted by downstream tools for metrics, traces still tend to be reliant on gRPC. If you are emitting the data in OTEL format Then we can receive it over HTTP or gRPC and send it really anywhere in that format - that includes passing it along to non-OTEL destinations. I have a couple of pipelines we've been honing for transforming OTEL into Cribl metrics format. There are some samples as well. I can pack that up and get it over to you so you can how it works. We've mostly been working on OTEL to cribl metrics and cribl metrics to OTEL. I've found some resources on Prom to OTEL but it's still a little all over the place due to the differences in how the metric types are handled in certain cases (counter being monotonically increasing in Prom and deltas in OTEL, differences in histograms). OTEL actually does a lousy job of producing examples/samples of payloads. I need to fire up my OTEL demo and grab some samples after walking the maniac dog and getting some dinner.

  • nicktank
    nicktank Posts: 26 mod

    JSON is what I've worked mostly with, not the binary format. But the protobuf schema is the same either way.

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    per some guys on the internet: https://www.timescale.com/blog/prometheus-vs-opentelemetry-metrics-a-complete-guide/ > In other words, Prometheus metrics are a strict subset of OpenTelemetry metrics.

  • nicktank
    nicktank Posts: 26 mod

    Was just reading that

  • Clint Sharp
    Clint Sharp Posts: 27 mod

    I'd love to get some resources to help you with this. <@U01C35EMQ01&gt; <@U021YQ08XS6&gt; <@U043J3H0EF3&gt;

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    So, since I'm scraping prom metrics using the cribl prometheus scraper, I figured I'm as close to native prom metrics as it gets? I understand the scraper is reading over HTTP, and once I get them in a pipeline, the events are very readable, but plaintext-y

  • nicktank
    nicktank Posts: 26 mod

    You'll see the same thing when receiving them from OTEL directly in stream - they'll appear in the protobuf schema but plain text-y. The work here is to convert from prom to otel schema and then use the otel dest to send and it should flow.

  • Clint Sharp
    Clint Sharp Posts: 27 mod

    yes, having a prom to otel pack would make sense

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    Yeah, I think I have a few teams that might be interested in routing through cribl instead of managing otel installations on all their devices, requiring more ansible/puppet/what-have-you

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    recall in my case I'm actually trying to send to a kafka destination configured to read otel collector output in protobuf, so destination flexibility would be valuable here as well :slightly_smiling_face:

  • nicktank
    nicktank Posts: 26 mod

    Prom --> Stream (OTEL conversion) ---> Kafka

  • nicktank
    nicktank Posts: 26 mod

    Where does it go from Kafka?

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    custom database for storing timeseries data

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    same lego block as the prometheus database. Think something grafana would read from

  • Clint Sharp
    Clint Sharp Posts: 27 mod

    <@U021YQ08XS6&gt; is on our tiger team that's working on these use cases and already attached to your account. I think we should setup a call.

  • Erin Sweeney
    Erin Sweeney Posts: 45 admin

    That would be awesome!

  • Franky Laarits
    Franky Laarits Posts: 59 ✭✭

    Perfect... <@U0319K5V9CM&gt;, when are you free? Any time Friday morning??