We have updated our Terms of Service, Code of Conduct, and Addendum.

Database as data source for lookup

Lukáš Mečíř
Lukáš Mečíř Posts: 6 ✭
in Stream

Hello,
I would like to ask one thing: is it possible to use some database (SQL etc.) as a data source for lookup? One of our customers has a database that contains data that he would like to use to enrich events processed on Cribl Stream. The idea is that he would read the data from the database into Cribl Stream using a suitable connector, save it as a CSV file and then use that as the source for a lookup to use in the pipeline.
Since I don't have experience with something like this yet, I would like to ask:
Is there anyone who has experience with this?
Is this a feasible way? And if so, is it suitable or are there better ones?
What is there to be careful about performance, load on Cribl and the database, etc.? (Enrichment would involve about 200 events per second, the database is in the lower tens of thousands of rows, and the data in the lookup should be refreshed once per hour.)
Any experience, ideas or recommendations would be greatly appreciated.

Thank you very much.

Lukas Mecir

Tagged:

Answers

  • Tony Reinke
    Tony Reinke Posts: 10 ✭

    Check out this talk from the first Criblcon:
    https://cribl.io/criblcon/enrichment-better-data-in-better-response-times-out/

    Also some docs:
    https://docs.cribl.io/stream/redis-function/

Categories