For bringing Amazon Kinesis Firehose in to Cribl, what format should I use?

mferris

Hi! I was looking into using cribl for sending in data from amazon kinesis firehose into cribl. But there is no format for the data to be send mentioned in the documentation. And there is no event breaker support also. So what format should be followed? is it ndjson or is there anything I'm missing?

Abby Strong

Hi <@U05ALUDRLD6&gt;, is this the documentation you have seen? https://docs.cribl.io/stream/sources-kinesis-firehose/

mferris

Yes. But there is no mention about the input data format

Abby Strong

Only this: ```This Source supports gzip-compressed inbound data when the Content‚ÄëEncoding: gzip connection header is set.```

Abby Strong

And, right at the beginning, it says that you should use Kinesis' HTTP endpoint destination option. Sorry, but I don't have experience with Kinesis Firehose. Is that information not adequate?

Michael Katz

Our Amazon Kinesis Firehose Source expects HTTP inbound data.

mferris

So we can pass any http inbound data format? (json, xml etc) <@U013R2W0L3S&gt;

Michael Katz

For that range, I too will need to defer to a Cribl colleague how has more practical experience ingesting from Firehose.

David Maislin

Any format, pick http or http_raw sources, define event breakers if needed, done.

David Maislin

NDJSON is fine for that.

mferris

Okkk thank you so much everyone