Does anyone have experience using cribl to send prometheus metrics?
Does anyone have experience using cribl to send prometheus metrics? In the OTEL destination doc, it describes that events must conform to otel protobuf spec, but doesn't describe how this would happen exactly: https://docs.cribl.io/stream/destinations-otel/#protocol-and-transport-support Is there any automatic translation that can be applied, either on the source, some pipeline function, or a destination?
Answers
-
So you want to take unstructured data and covert it to an OTEL compatible format as metrics?
0 -
In this case, the metrics I'm receiving are generated by prometheus node exporter, so they should be nicely convertible
0 -
This can be done and I am about to implment this exact scenario(stream source `prometheus scraper` beautifully scrapes node exporter metrics and then further you can send it wherever(we are about to send it to a splunk metrics index). The reason we are doing this is just for 2 nodes is, these are facing the big bad internet and my infosec team insists that I can't get an agent like telegraf to push the data out to cribl stream(which is in a relatively safer network). Is there a specific reason why you are planning such an approach?
0 -
In my case I have some resource/security constrained hosts where I wouldn't want to install an otel collecter directly, but instead want to use cribl as a replacement since it's distributed, has queueing, etc.
0 -
We have looked into this before. It's hard fwiw. Otel schema is very complex.
0 -
<@U01J91JRGL9> we had talked about documenting the schema that would be required in order to get that out? Do we have a JSON Schema that could be used for validation?
0 -
I know only so much about protobuf messaging, but it seems like the otel metrics.proto is reasonably straightforward: https://github.com/open-telemetry/opentelemetry-proto/blob/v0.20.0/opentelemetry/proto/metrics/v1/metrics.proto Does cribl already implement this? If not, is there a way I can install protobufjs and write a script function in a pipeline to do the encoding myself?
0 -
Yes, we implement this. There are two protocols for Otel, one that uses JSON over HTTP and one that uses Protobuf. I believe we support both now.
0 -
If you match the schema it should just flow.
0 -
I was looking for something in the OTEL destination config that indicated the encoding it would use, but then the docs mentioned it had to be already properly encoded before sending to the destination.
0 -
My understanding is that if I were using the OTEL collector, I would have other options for where to send my now-protobuf-formatted data as well. So in my case, I'm actually hoping to format the metrics in protobuf as though they were produced by an OTEL collector, but actually send the data to Kafka
0 -
It uses Protobuf by default, but that's an implementation detail. Here are some JSON Schemas that can be used for validation (I think): https://github.com/open-telemetry/opentelemetry-configuration/tree/main/schema
0 -
HTTP is overwhelmingly the protocol we see being adopted by downstream tools for metrics, traces still tend to be reliant on gRPC. If you are emitting the data in OTEL format Then we can receive it over HTTP or gRPC and send it really anywhere in that format - that includes passing it along to non-OTEL destinations. I have a couple of pipelines we've been honing for transforming OTEL into Cribl metrics format. There are some samples as well. I can pack that up and get it over to you so you can how it works. We've mostly been working on OTEL to cribl metrics and cribl metrics to OTEL. I've found some resources on Prom to OTEL but it's still a little all over the place due to the differences in how the metric types are handled in certain cases (counter being monotonically increasing in Prom and deltas in OTEL, differences in histograms). OTEL actually does a lousy job of producing examples/samples of payloads. I need to fire up my OTEL demo and grab some samples after walking the maniac dog and getting some dinner.
0 -
JSON is what I've worked mostly with, not the binary format. But the protobuf schema is the same either way.
0 -
per some guys on the internet: https://www.timescale.com/blog/prometheus-vs-opentelemetry-metrics-a-complete-guide/ > In other words, Prometheus metrics are a strict subset of OpenTelemetry metrics.
0 -
Was just reading that
0 -
I'd love to get some resources to help you with this. <@U01C35EMQ01> <@U021YQ08XS6> <@U043J3H0EF3>
0 -
So, since I'm scraping prom metrics using the cribl prometheus scraper, I figured I'm as close to native prom metrics as it gets? I understand the scraper is reading over HTTP, and once I get them in a pipeline, the events are very readable, but plaintext-y
0 -
You'll see the same thing when receiving them from OTEL directly in stream - they'll appear in the protobuf schema but plain text-y. The work here is to convert from prom to otel schema and then use the otel dest to send and it should flow.
0 -
yes, having a prom to otel pack would make sense
0 -
Yeah, I think I have a few teams that might be interested in routing through cribl instead of managing otel installations on all their devices, requiring more ansible/puppet/what-have-you
0 -
recall in my case I'm actually trying to send to a kafka destination configured to read otel collector output in protobuf, so destination flexibility would be valuable here as well :slightly_smiling_face:
0 -
Prom --> Stream (OTEL conversion) ---> Kafka
0 -
Where does it go from Kafka?
0 -
custom database for storing timeseries data
0 -
same lego block as the prometheus database. Think something grafana would read from
0 -
<@U021YQ08XS6> is on our tiger team that's working on these use cases and already attached to your account. I think we should setup a call.
0 -
That would be awesome!
0 -
Perfect... <@U0319K5V9CM>, when are you free? Any time Friday morning??
0
