We have updated our Terms of Service, Code of Conduct, and Addendum.

S3/Blob storage folder selection as source

Options

Hi Team,

I am a Cribl Certified Engineer and have been working on a use case which is described below, but I haven't achieved the expected output.

The use case involves an S3/Blob Storage where we have one bucket containing 2-3 folders. The source should be a specific folder, and from that folder, I need to ingest data to the required destination.

In S3/Blob, I can see that regex can be used for matching file names to download and process, with the default being .*. Can I change it to /foldername/.*? Will this work? If yes, can I have a whitepaper or some documentation on this?

Tagged: