S3/Blob storage folder selection as source

Venkatesh Panchariya

Hi Team,

I am a Cribl Certified Engineer and have been working on a use case which is described below, but I haven't achieved the expected output.

The use case involves an S3/Blob Storage where we have one bucket containing 2-3 folders. The source should be a specific folder, and from that folder, I need to ingest data to the required destination.

In S3/Blob, I can see that regex can be used for matching file names to download and process, with the default being .*. Can I change it to /foldername/.*? Will this work? If yes, can I have a whitepaper or some documentation on this?