Hello,
Here I have a small picture of how the environment is structured:
Red arrow -> Source Splunk TCP (Cribl Stream)
I'm trying to forward the journald data from the Splunk Universal Forwarder to the Cribl Worker (Black to blue box).
I have configured the forwarding of the journald data using the instructions from Splunk.
(Get data with the Journald input - Splunk Documentation)
I can forward the journald data and it also arrives at the cribl worker.
Problem: the cribl worker cannot distinguish the individual events from the journald data or does not know when a single event is over and thus combines several individual events into one large one.
The Cribl Worker always merges about 5-8 journald events.
(I have marked the individual events here. However, they arrive as such a block, sometimes more together, sometimes less.)
Event 1:
Invalid user test from 111.222.333.444port 1111pam_unix(sshd:auth):check pass; userunknownpam_unix(sshd:auth):authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.222.333.444Failed password forinvalid user testfrom 111.222.333.444 port1111 ssh2error: Received disconnect from 111.222.333.444port 1111:13: Unableto authenticate [preauth]Disconnected from invaliduser test 111.222.333.444port 1111 [preauth]
What I tested:
If I have the journald data from the universal forwarder not forwarded via a cribl worker, but via a heavy forwarder (The blue box in the picture above is then no longer a Cribl Worker but a Splunk Heavy Forwarder), then the events are individual and easy to read. Like this:
Event 1:
Invalid user testfrom 111.222.333.444 port1111
Event 2:
pam_unix(sshd:auth):check pass; userunknown
Event 3:
pam_unix(sshd:auth):authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.222.333.444
Event 4:
Failed password forinvalid user testfrom 111.222.333.444 port1111 ssh2
Event 5:
error: Received disconnectfrom 111.222.333.444 port1111:13: Unable toauthenticate [preauth]
Event 6:
Disconnected from invaliduser test 111.222.333.444port 1111 [preauth]
--------------------------------
I'm looking for a solution that I can send the journald data as shown in the figure above, but the journald data will be sent as in the second case.
Thanks in advance for your help.
Unfortunately, as you're introducing an additional external component (cribl worker), it's hard to say what happens where.
BTW, it's probably not that cribl merges anything, more like it doesn't split the events properly since UF sends data in chunks, not single events.
So the one at fault here is most probably the cribl one. BTW, why don't you just send UF->Indexer (or UF->HF->Indexer)?