We have updated our Terms of Service, Code of Conduct, and Addendum.

Forwarding data from On-Prem Workers to Hybrid Cloud

Steve Settles
Steve Settles Posts: 2 ✭
edited September 2023 in Stream

Hello Cribl gurus - could you square https://cribl.io/blog/how-to-architect-your-logstream-to-logstream-data-flows/ with https://docs.cribl.io/stream/destinations-cribl-tcp/#reqs & https://docs.cribl.io/stream/destinations-cribl-http#reqs? We have an on-prem distributed deployment and a Cloud hybrid deployment, and we would like to forward data from one source from the on-prem workers to a hybrid Cloud deployment to continue our Cribl PS engagement, but it’s unclear if this is possible and if so how to achieve it. To be clear, our on-prem and cloud instances/deployments’ have different tokens. Please advise what options, if any, we have to connect the 2 instances together in terms of the on-prem source of interest.

image.png

Tagged:

Best Answer

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod
    Answer ✓

    To fully leverage Cribl TCP and Cribl HTTP the sender and receiver groups must be managed by the same leader. Otherwise the events will be double counted against licensing. By being managed by the same leader they would also have the same token. You may as well use TCP JSON which also uses tokens but that is a more generic protocol which can be used when you'd be double counted with the license anyway because of the sender/receiver not being in the same cluster to leverage the benefits of Cribl TCP and HTTP.

Answers

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod
    Answer ✓

    To fully leverage Cribl TCP and Cribl HTTP the sender and receiver groups must be managed by the same leader. Otherwise the events will be double counted against licensing. By being managed by the same leader they would also have the same token. You may as well use TCP JSON which also uses tokens but that is a more generic protocol which can be used when you'd be double counted with the license anyway because of the sender/receiver not being in the same cluster to leverage the benefits of Cribl TCP and HTTP.

  • Steve Settles
    Steve Settles Posts: 2 ✭

    so https://cribl.io/blog/how-to-architect-your-logstream-to-logstream-data-flows/ assumes leaders are part of the same instance/deployment?

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod

    I wrote that blog post before cribl tcp and http existed. The protocols discussed in there don't have the restriction of sender/receiver needing to be managed by the same leader.

Categories