For bringing Amazon Kinesis Firehose in to Cribl, what format should I use?

mferris

Hi! I was looking into using cribl for sending in data from amazon kinesis firehose into cribl. But there is no format for the data to be send mentioned in the documentation. And there is no event breaker support also. So what format should be followed? is it ndjson or is there anything I'm missing?

mferris

Yes. But there is no mention about the input data format

Abby Strong

Hi <@U05ALUDRLD6&gt;, is this the documentation you have seen? https://docs.cribl.io/stream/sources-kinesis-firehose/

Abby Strong

Only this: ```This Source supports gzip-compressed inbound data when the Content‚ÄëEncoding: gzip connection header is set.```

Abby Strong

And, right at the beginning, it says that you should use Kinesis' HTTP endpoint destination option. Sorry, but I don't have experience with Kinesis Firehose. Is that information not adequate?

mferris

So we can pass any http inbound data format? (json, xml etc) <@U013R2W0L3S&gt;

Michael Katz

Our Amazon Kinesis Firehose Source expects HTTP inbound data.

David Maislin

Any format, pick http or http_raw sources, define event breakers if needed, done.

Michael Katz

For that range, I too will need to defer to a Cribl colleague how has more practical experience ingesting from Firehose.

mferris

Okkk thank you so much everyone

David Maislin

NDJSON is fine for that.