Can I aggregate events multiple collector tasks from a single collection job into a single event?
I run multiple REST collectors where the APIs I am collecting from have hard limits on the number of events returned which requires me to paginate.
Doing this creates multiple collection tasks. Part of my main pipeline runs a custom REST function to create an object in the API based on a field in the event and returns data to enrich my event. If the values exists in the API I get an "this already exists" error. I normally just aggregate this field, make my API call, then join and unroll the events on the enrichments.
This breaks when the REST collector creates multiple collection tasks as it seems that I cannot aggregate all of my events into one single aggregate event even when all the events are within my time window.
How can I aggregate my events from a single job that creates multiple collection tasks into a single event?