We have updated our Terms of Service, Code of Conduct, and Addendum.

Error with Google Cloud Storage Destination

JH Liew
JH Liew Posts: 13 ✭
edited September 2023 in Stream

Hi, I am currently exploring a way to collect event data and looking to save it into a temporary staging storage (GCS Bucket) before transferring the data downstream for further processing.

I am having difficulty to get a working configuration on Stream no matter what I tried. Stream worker is always throwing the following error: "failed to close file" where the detailed message is saying "Invalid argument." without much details except that the file that Stream worker is having issue to close is at this location: /opt/cribl/state/outputs/staging/prefix/Bucket_yyyymmdd_hhmm/yyyy/mm/dd/CriblOut-<random>.0.json.tmp

I have checked that the credentials for accessing the bucket is valid and the permissions for the given credentials are alright for upload files into the bucket.

Does anyone have any idea how I might be able to get around this problem?

Thanks.

Tagged:

Answers

  • Tomer Shvueli
    Tomer Shvueli Posts: 17 mod

    Hi @JH Liew ,

    What version of Stream are you running? Does the user running Stream have file permissions to access and remove files in the staging path that you put?

  • JH Liew
    JH Liew Posts: 13 ✭

    This was resolved some time ago and I've completely forgotten to post back. It appeared that the issue was the GCS storage not set correctly with Fine-grained Access Control. Once that GCS bucket has been configured with it, the issue went away.

Categories