Import this field list and throw away all fields in the data that are not contained in the list

Tony Reinke - Cribl

Hi everyone,

I have a use case atm where I have a lookup that contains a column of "allowed fields" for various sourcetypes, that is, the value for a given sourcetype is a common separted list of field names. At the end of my pipeline, I basically want to import this field list and throw away all fields in the data that are not contained in the list (because I'm finished with the normalization). Is there an easy way to do this? The serialize function has essentially the same functionality, when you import the parser you get a keep fields list. But how to do this without the parser?

Original message in Cribl Community Slack Message: https://cribl-community.slack.com/archives/CPYBPK65V/p1682331204917309

Ralph Nowitzki

It could make this work with a Code Function.

Created a Lookup with 2 columns: sourcetype and fields
In that lookup 1 row: "test_source" and "test1,test2"

Created a sample event with different fields.
Amongst other fields : "test1" and "test3" and "sourcetype" (= "test_source")

In the pipeline created a field called "fields" and used an eval to get the field list from the Lookup:
C.Lookup('sourcetype_field_filter_test.csv', 'sourcetype').match(sourcetype, 'fields')


Now the field called "fields" has the value "test1,test2".

With the following Code function all fields besides the one in the "fields" field are removed (cribl_pipe remains/is added at the end)

try {   for (let [key, value] of Object.entries(__e)) {      if (!__e.fields.includes(key)) {          __e[key] = undefined       }   }} catch (err) {   __e.CRIBLERR = err}

= If the field name is within the "fields" array, eval the fieldname to undefined (remove it).

In my case only test1 was left. test3 and all other fields were gone.

Ralph Nowitzki

It could make this work with a Code Function.

Created a Lookup with 2 columns: sourcetype and fields
In that lookup 1 row: "test_source" and "test1,test2"

Created a sample event with different fields.
Amongst other fields : "test1" and "test3" and "sourcetype" (= "test_source")

In the pipeline created a field called "fields" and used an eval to get the field list from the Lookup:
C.Lookup('sourcetype_field_filter_test.csv', 'sourcetype').match(sourcetype, 'fields')


Now the field called "fields" has the value "test1,test2".

With the following Code function all fields besides the one in the "fields" field are removed (cribl_pipe remains/is added at the end)

try {   for (let [key, value] of Object.entries(__e)) {      if (!__e.fields.includes(key)) {          __e[key] = undefined       }   }} catch (err) {   __e.CRIBLERR = err}

= If the field name is within the "fields" array, eval the fieldname to undefined (remove it).

In my case only test1 was left. test3 and all other fields were gone.