Is there any info on Search costs when searching AWS S3 buckets?

Erin Sweeney

Is there any info on Search costs when searching AWS S3 buckets?   Specifically on how data transfer works.  For example, does Search hit S3 from US East and is $.01 per GB against my AWS account?  Is Search caching at all? https://aws.amazon.com/s3/pricing/?p=pm&c=s3&z=4


Original post was from https://cribl-community.slack.com/archives/C03HK1K33FA/p1678133798883439

Brendan Dalpe

On the AWS side, you pay for API calls to the bucket. Assuming the bucket is in the same region as the Cribl.Cloud instance, there is no data transfer fee. Assuming you're using S3 standard storage tier, the API calls we would be making:
LIST: $0.005/1000 requests
GET: $0.0004/1000 requests

Erin Sweeney

My region via the overview page says us-west-2.  Does this mean it is to my advantage to host my s3 in us-west-2?

Brendan Dalpe

Yes, otherwise you're paying extra money to search that bucket.

Erin Sweeney

Okay, thanks.  Also, are there any logs showing the buckets and/or objects being hit from Search?  I only saw logs from stream, edge, etc.

David Cavuto

S3 access logs should be visible in the Search Status page, and when searching the internal Search logs themselves

Chris Breshears

Use case you mention above,  if I am not searching, will I get charged api hits? consistantly?

David Cavuto

you only get charged API hits when you actively run a search. There's no background or periodic updates that occur at this time.

Chris Breshears

This definitely helps with understanding how it works, thanks! 

David Cavuto

we've actually added to the docs some information for clarity here: https://docs.cribl.io/search/set-up-s3#data-transfer-charges