We have updated our Terms of Service, Code of Conduct, and Addendum.

Import this field list and throw away all fields in the data that are not contained in the list

Tony Reinke - Cribl
Tony Reinke - Cribl Posts: 134 admin
edited July 12 in Stream

Hi everyone,

I have a use case atm where I have a lookup that contains a column of "allowed fields" for various sourcetypes, that is, the value for a given sourcetype is a common separted list of field names. At the end of my pipeline, I basically want to import this field list and throw away all fields in the data that are not contained in the list (because I'm finished with the normalization). Is there an easy way to do this? The serialize function has essentially the same functionality, when you import the parser you get a keep fields list. But how to do this without the parser?

Original message in Cribl Community Slack Message: https://cribl-community.slack.com/archives/CPYBPK65V/p1682331204917309

Best Answer

  • Ralph Nowitzki
    Ralph Nowitzki Posts: 6
    Answer ✓

    It could make this work with a Code Function.

    Created a Lookup with 2 columns: sourcetype and fields
    In that lookup 1 row: "test_source" and "test1,test2"

    Created a sample event with different fields.
    Amongst other fields : "test1" and "test3" and "sourcetype" (= "test_source")

    In the pipeline created a field called "fields" and used an eval to get the field list from the Lookup:
    C.Lookup('sourcetype_field_filter_test.csv', 'sourcetype').match(sourcetype, 'fields')

    Now the field called "fields" has the value "test1,test2".

    With the following Code function all fields besides the one in the "fields" field are removed (cribl_pipe remains/is added at the end)

    try {   for (let [key, value] of Object.entries(__e)) {      if (!__e.fields.includes(key)) {          __e[key] = undefined       }   }} catch (err) {   __e.CRIBLERR = err}

    = If the field name is within the "fields" array, eval the fieldname to undefined (remove it).

    In my case only test1 was left. test3 and all other fields were gone.

Answers

  • Ralph Nowitzki
    Ralph Nowitzki Posts: 6
    Answer ✓

    It could make this work with a Code Function.

    Created a Lookup with 2 columns: sourcetype and fields
    In that lookup 1 row: "test_source" and "test1,test2"

    Created a sample event with different fields.
    Amongst other fields : "test1" and "test3" and "sourcetype" (= "test_source")

    In the pipeline created a field called "fields" and used an eval to get the field list from the Lookup:
    C.Lookup('sourcetype_field_filter_test.csv', 'sourcetype').match(sourcetype, 'fields')

    Now the field called "fields" has the value "test1,test2".

    With the following Code function all fields besides the one in the "fields" field are removed (cribl_pipe remains/is added at the end)

    try {   for (let [key, value] of Object.entries(__e)) {      if (!__e.fields.includes(key)) {          __e[key] = undefined       }   }} catch (err) {   __e.CRIBLERR = err}

    = If the field name is within the "fields" array, eval the fieldname to undefined (remove it).

    In my case only test1 was left. test3 and all other fields were gone.