Cribl Stream - Collect journalctl events with a Splunk UF to Cribl Stream in individual events
![Jonas Ehrenmann](https://s.gravatar.com/avatar/ac417428ff755f14a682aa3f75a8a5b7?s=480&r=pg&d=https%3A%2F%2Fcdn.auth0.com%2Favatars%2Fjo.png)
Hello,
Here I have a small picture of how the environment is structured:
Red arrow -> Source Splunk TCP (Cribl Stream)
I'm trying to forward the journald data from the Splunk Universal Forwarder to the Cribl Worker (Black to blue box).
I have configured the forwarding of the journald data using the instructions from Splunk.
(Get data with the Journald input - Splunk Documentation)
I can forward the journald data and it also arrives at the cribl worker.
Problem: the cribl worker cannot distinguish the individual events from the journald data or does not know when a single event is over and thus combines several individual events into one large one.
The Cribl Worker always merges about 5-8 journald events.
(I have marked the individual events here. However, they arrive as such a block, sometimes more together, sometimes less.)
Event 1:
Invalid user test from 111.222.333.444port 1111pam_unix(sshd:auth):check pass; userunknownpam_unix(sshd:auth):authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.222.333.444Failed password forinvalid user testfrom 111.222.333.444 port1111 ssh2error: Received disconnect from 111.222.333.444port 1111:13: Unableto authenticate [preauth]Disconnected from invaliduser test 111.222.333.444port 1111 [preauth]
What I tested:
If I have the journald data from the universal forwarder not forwarded via a cribl worker, but via a heavy forwarder (The blue box in the picture above is then no longer a Cribl Worker but a Splunk Heavy Forwarder), then the events are individual and easy to read. Like this:
Event 1:
Invalid user testfrom 111.222.333.444 port1111
Event 2:
pam_unix(sshd:auth):check pass; userunknown
Event 3:
pam_unix(sshd:auth):authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.222.333.444
Event 4:
Failed password forinvalid user testfrom 111.222.333.444 port1111 ssh2
Event 5:
error: Received disconnectfrom 111.222.333.444 port1111:13: Unable toauthenticate [preauth]
Event 6:
Disconnected from invaliduser test 111.222.333.444port 1111 [preauth]
------------------------------------------------
I'm looking for a solution that I can send the journald data as shown in the figure above, but the journald data will be sent as in the second case.
Thanks in advance for your help.
Comments
-
It looks like you'll need to build a custom event break. If you can send a sample of the log I can break it up for you if you know the pattern of the log. The issue with what I see here is there isn't a very predictable pattern for it but we can take a whack at it.
If you go to Processing → Knowledge → Event Breaker Rules
Create a breaker and go to add rule
Copy and paste your sample and try to build your custom event break. Personally I would have to look up more on how journald write events to file. I think its a newline per event.
And then build your regex
If I have some free time today I'll look at journald and see what I can do to help out more.
0