We have updated our Terms of Service, Code of Conduct, and Addendum.

Cribl on AWS Fargate

Joel Yue
Joel Yue Posts: 4 ✭
edited November 2023 in Stream

How do i calculate the amount of workers i need if i am using AWS Fargate arm processors? I am looking at 7tb/day input and output to 2 destination which is a total of 21tb/day throughput.

Thanks

Tagged:

Answers

  • Justin Hamblin
    Justin Hamblin Posts: 7 ✭

    Hello @Joel Yue!

    Weather you are running Cribl on a VM or Serverless, CPU and Memory resources requirements remain the same. You mentioned Araviton processors, and we've got some documentation on this processor type here.

    The calculation (IN + OUT = X gb/day) helps us estimate the resources required to process your workload. This calculation largely is universal reguardless of how you choose to run Cribl.

    The primary differences (between VM's and Serverless deployments) to keep in mind are;

    1. When leveraging a VM deployment, we recommend reserving 2 cpu cores for normal OS level activities. The remaining resources can be used by Cribl processes. This reservation concept isn't important in a container deployment because a container is the process or collection of processes.
      1. https://docs.cribl.io/stream/deploy-architecture#
    2. When leveraging Serverless/Containters, Make sure you assign persistent storage IF you wish to leverage Persistent Queuing.
      1. https://docs.cribl.io/stream/deploy-kubernetes#using-persistent-storage-for-persistent-queueing

    With that siad, you can always leverage the online calculator to get you in the ballpark of resource requirments. As with any sizing exercise, its best to reachout to your sales team to ensure you have the support you need to size your deployment.

Categories