Cribl Stream - Collect journalctl events with a Splunk UF to Cribl Stream in individual events

Jonas Ehrenmann

Hello,

Here I have a small picture of how the environment is structured:

Red arrow -> Source Splunk TCP (Cribl Stream)

I'm trying to forward the journald data from the Splunk Universal Forwarder to the Cribl Worker (Black to blue box).

I have configured the forwarding of the journald data using the instructions from Splunk.

(Get data with the Journald input - Splunk Documentation)

I can forward the journald data and it also arrives at the cribl worker.

Problem: the cribl worker cannot distinguish the individual events from the journald data or does not know when a single event is over and thus combines several individual events into one large one.

The Cribl Worker always merges about 5-8 journald events.

(I have marked the individual events here. However, they arrive as such a block, sometimes more together, sometimes less.)

Event 1:

Invalid user test from 111.222.333.444port 1111pam_unix(sshd:auth):check pass; userunknownpam_unix(sshd:auth):authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.222.333.444Failed password forinvalid user testfrom 111.222.333.444 port1111 ssh2error: Received disconnect from 111.222.333.444port 1111:13: Unableto authenticate [preauth]Disconnected from invaliduser test 111.222.333.444port 1111 [preauth]

What I tested:

If I have the journald data from the universal forwarder not forwarded via a cribl worker, but via a heavy forwarder (The blue box in the picture above is then no longer a Cribl Worker but a Splunk Heavy Forwarder), then the events are individual and easy to read. Like this:

Event 1:

Invalid user testfrom 111.222.333.444 port1111

Event 2:

pam_unix(sshd:auth):check pass; userunknown

Event 3:

pam_unix(sshd:auth):authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.222.333.444

Event 4:

Failed password forinvalid user testfrom 111.222.333.444 port1111 ssh2

Event 5:

error: Received disconnectfrom 111.222.333.444 port1111:13: Unable toauthenticate [preauth]

Event 6:

Disconnected from invaliduser test 111.222.333.444port 1111 [preauth]

------------------------------------------------

I'm looking for a solution that I can send the journald data as shown in the figure above, but the journald data will be sent as in the second case.

Thanks in advance for your help.

nthusiast

It looks like you'll need to build a custom event break. If you can send a sample of the log I can break it up for you if you know the pattern of the log. The issue with what I see here is there isn't a very predictable pattern for it but we can take a whack at it.

If you go to Processing → Knowledge → Event Breaker Rules

Create a breaker and go to add rule

Copy and paste your sample and try to build your custom event break. Personally I would have to look up more on how journald write events to file. I think its a newline per event.

And then build your regex

If I have some free time today I'll look at journald and see what I can do to help out more.