For bringing Amazon Kinesis Firehose in to Cribl, what format should I use?
Hi! I was looking into using cribl for sending in data from amazon kinesis firehose into cribl. But there is no format for the data to be send mentioned in the documentation. And there is no event breaker support also. So what format should be followed? is it ndjson or is there anything I'm missing?
Answers
-
Hi <@U05ALUDRLD6>, is this the documentation you have seen? https://docs.cribl.io/stream/sources-kinesis-firehose/
0 -
Yes. But there is no mention about the input data format
0 -
Only this: ```This Source supports gzip-compressed inbound data when the Content‚ÄëEncoding: gzip connection header is set.```
0 -
And, right at the beginning, it says that you should use Kinesis' HTTP endpoint destination option. Sorry, but I don't have experience with Kinesis Firehose. Is that information not adequate?
0 -
Our Amazon Kinesis Firehose Source expects HTTP inbound data.
0 -
So we can pass any http inbound data format? (json, xml etc) <@U013R2W0L3S>
0 -
For that range, I too will need to defer to a Cribl colleague how has more practical experience ingesting from Firehose.
0 -
Any format, pick http or http_raw sources, define event breakers if needed, done.
0 -
NDJSON is fine for that.
0 -
Okkk thank you so much everyone
0