We have updated our Terms of Service, Code of Conduct, and Addendum.

Data flow between stages

Sven Breier
Sven Breier Posts: 14 ✭
in Stream

Hello,
i need some input.

We want to have a separate dataflow between different stages within cribl.
So we receive data in our prod environment and want to have an extra dataflow to QS stage, to test parsing changes etc.
One way we know is to configure a second output wihtin “Output Router”
Are there any other ways? Maybe within routes or pipelines?

Thank you for your input

Best Answer

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod
    Answer ✓

    You could configure two (or more) routes with the same filter. The first routes should have the final flag disabled. The last route should have the flag set to final. This way you can copy the same data stream to multiple destinations without having to make changes to the destination frequently.

    A trick here is to also include ... && Math.random() < 0.1 to your route filter to sample the number of events matching and being forwarded to the destination. You could also look into the Sample, Dynamic Sampling, Suppression, and Drop functions if you wanted to reduce the volume in the pipeline.

Answers

  • Jon Rust
    Jon Rust Posts: 475 mod

    Also check out the recent blog post: https://cribl.io/blog/bifurcating-observability-data-to-multiple-destinations/

  • Sven Breier
    Sven Breier Posts: 14 ✭

    Thank you,
    Is there any way to just configure it via pipelines and without destinations?
    The best way for us is to deploy it without and restart of the workers.

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod
    Answer ✓

    You could configure two (or more) routes with the same filter. The first routes should have the final flag disabled. The last route should have the flag set to final. This way you can copy the same data stream to multiple destinations without having to make changes to the destination frequently.

    A trick here is to also include ... && Math.random() < 0.1 to your route filter to sample the number of events matching and being forwarded to the destination. You could also look into the Sample, Dynamic Sampling, Suppression, and Drop functions if you wanted to reduce the volume in the pipeline.

  • Sven Breier
    Sven Breier Posts: 14 ✭

    Thank you for your thoughts and inputs.
    Well test the last one from @bdalpe which seems to be the best way for us.

  • Shane Daniels
    Shane Daniels Posts: 28 mod
    edited July 2023

    If you are looking to have data come into Cribl and then have the option to send it out in multiple formats to various destinations, you are most likely looking for a Post processing pipeline. That way you can work on the data in a consistent way and then format for each downstream system appropriately.

    docs.cribl.io

    Pipelines | Cribl Docs

    How do Pipelines work, settings and types

    1c5b10e6f195f826697c720be90619628a228761.png

Categories