Patterns for suppressing Splunk cooked data

hpapazov

Is there a pattern that I can use to suppress Splunk logs based on data that is part of the _raw field? So far I have come up with a pipeline that uses Regex Extract, Suppress, and Eval function. I am extracting the key:value pair that I need from the _raw field, utilize it in my suppression function, and then remove it from the logs before sending the event to Splunk. While this seems to be working, I wonder if there is a better way to achieve this especially if we want to add additional key:value pairs to suppress on in the future? I was looking into the Parser function that allows me to extract all key:value pairs from the _raw field but I could not find a way to remove all those from the event before sending it to Splunk.

Jon Rust

Hold on, i misread your post

Use the Parser function to Extract _raw into a new field, maybe __myfield. Now you can refer to __myfield in your suppression action.

Jon Rust

Can you supply a sample log? Absolutely we can add/remove/modify fields in any part of the event. Id just like to see real world example before getting too deep into particulars.

hpapazov

Here is an example log that I see in Cribl. Just FYI, I replaced the values for some of the fields because that info is internal to the company. So based on the below example, I am trying to find a way to be able to suppress on any of the key:value pairs that are part of the _raw field.

{
    "_raw": "{\"Timestamp\":\"2022-05-03T14:51:25.0606688+00:00\",\"Level\":\"Information\",\"MessageTemplate\":\"Successfully retrieved data for {service}\",\"RenderedMessage\":\"Successfully retrieved data for MyService\",\"Properties\":{\"count\":0,\"WorkspaceID\":1234567,\"SourceContext\":\"MyService.Worker.DataRetriever\",\"Priority\":2,\"InstanceIdentifier\":\"c783c624-d5c5-4eb7-a08c-9f50634b08e4\",\"DocumentID\":1234567,\"CorrelationID\":\"ff36e36b-ce29-4cad-9483-c2864dbe84aa\",\"HostName\":\"my-host\",\"ImageTag\":\"v1.0.290\",\"System\":\"MyService\",\"ProcessID\":19056,\"Application\":\"2.224.0\",\"SubSystem\":\"Worker\"}}",
    "_time": 1651589485.060668,
    "source": "2.224.0",
    "host": "10.10.1-.100",
    "sourcetype": "MyService",
    "index": "myIndex",
    "timestartpos": "14",
    "timeendpos": "47",
    "date_second": "25",
    "date_hour": "14",
    "date_minute": "51",
    "date_year": "2022",
    "date_month": "may",
    "date_mday": "3",
    "date_wday": "tuesday",
    "date_zone": "0",
    "punct": "{\"\":\"--::.+:\",\"\":\"\",\"\":\"__{}____{},__{}\",\"\":\"_____",
    "MessageTemplate": "\",
    "cribl_pipe": "suppression-test"
  }

Jon Rust

Hold on, i misread your post

Use the Parser function to Extract _raw into a new field, maybe __myfield. Now you can refer to __myfield in your suppression action.

hpapazov

I was just about to post that we found out that we can do what you are suggesting Jon. I think that does what we need. Thanks for the help Jon!

Jon Rust

Excellent! The trick is making sure your JSON objects are parsed before you start trying to manipulate the contents. In the preview window, if they have an a next to them, it means theyre a string. Only string functions will work. But if they have a {} next to them, its seen as an object and ready to be pulled apart like a lego kit. You can use the Parser function, or JSON.parse() in an Eval to parse a string that is proper JSON into an object.