We have updated our Terms of Service, Code of Conduct, and Addendum.

Cribl Stream - Collect journalctl events with a Splunk UF to Cribl Stream in individual events

Jonas Ehrenmann
Jonas Ehrenmann Posts: 1
edited June 4 in Stream

Hello,

Here I have a small picture of how the environment is structured:

Red arrow -> Source Splunk TCP (Cribl Stream)

I'm trying to forward the journald data from the Splunk Universal Forwarder to the Cribl Worker (Black to blue box).

I have configured the forwarding of the journald data using the instructions from Splunk.

(Get data with the Journald input - Splunk Documentation)

I can forward the journald data and it also arrives at the cribl worker.

Problem: the cribl worker cannot distinguish the individual events from the journald data or does not know when a single event is over and thus combines several individual events into one large one.

The Cribl Worker always merges about 5-8 journald events.

(I have marked the individual events here. However, they arrive as such a block, sometimes more together, sometimes less.)

Event 1:

Invalid user test from 111.222.333.444port 1111pam_unix(sshd:auth):check pass; userunknownpam_unix(sshd:auth):authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.222.333.444Failed password forinvalid user testfrom 111.222.333.444 port1111 ssh2error: Received disconnect from 111.222.333.444port 1111:13: Unableto authenticate [preauth]Disconnected from invaliduser test 111.222.333.444port 1111 [preauth]

What I tested:

If I have the journald data from the universal forwarder not forwarded via a cribl worker, but via a heavy forwarder (The blue box in the picture above is then no longer a Cribl Worker but a Splunk Heavy Forwarder), then the events are individual and easy to read. Like this:

Event 1:

Invalid user testfrom 111.222.333.444 port1111

Event 2:

pam_unix(sshd:auth):check pass; userunknown

Event 3:

pam_unix(sshd:auth):authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.222.333.444

Event 4:

Failed password forinvalid user testfrom 111.222.333.444 port1111 ssh2

Event 5:

error: Received disconnectfrom 111.222.333.444 port1111:13: Unable toauthenticate [preauth]

Event 6:

Disconnected from invaliduser test 111.222.333.444port 1111 [preauth]

------------------------------------------------

I'm looking for a solution that I can send the journald data as shown in the figure above, but the journald data will be sent as in the second case.

Thanks in advance for your help.

Comments

  • nthusiast
    nthusiast Posts: 6
    edited June 6

    It looks like you'll need to build a custom event break. If you can send a sample of the log I can break it up for you if you know the pattern of the log. The issue with what I see here is there isn't a very predictable pattern for it but we can take a whack at it.

    If you go to Processing → Knowledge → Event Breaker Rules

    Create a breaker and go to add rule

    Copy and paste your sample and try to build your custom event break. Personally I would have to look up more on how journald write events to file. I think its a newline per event.

    And then build your regex

    If I have some free time today I'll look at journald and see what I can do to help out more.