Getting Data In

Collect journalctl events with a Splunk UF to Cribl Stream in individual events

Uzumaki
Explorer

Hello,

Here I have a small picture of how the environment is structured:

test.png

Red arrow -> Source Splunk TCP (Cribl Stream)

 

I'm trying to forward the journald data from the Splunk Universal Forwarder to the Cribl Worker (Black to blue box).

I have configured the forwarding of the journald data using the instructions from Splunk.

(Get data with the Journald input - Splunk Documentation)

 

I can forward the journald data and it also arrives at the cribl worker.

Problem: the cribl worker cannot distinguish the individual events from the journald data or does not know when a single event is over and thus combines several individual events into one large one.

The Cribl Worker always merges about 5-8 journald events.

(I have marked the individual events here. However, they arrive as such a block, sometimes more together, sometimes less.)

Event 1:

Invalid user test from 111.222.333.444port 1111pam_unix(sshd:auth):check pass; userunknownpam_unix(sshd:auth):authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.222.333.444Failed password forinvalid user testfrom 111.222.333.444 port1111 ssh2error: Received disconnect from 111.222.333.444port 1111:13: Unableto authenticate [preauth]Disconnected from invaliduser test 111.222.333.444port 1111 [preauth]

 

What I tested:

If I have the journald data from the universal forwarder not forwarded via a cribl worker, but via a heavy forwarder (The blue box in the picture above is then no longer a Cribl Worker but a Splunk Heavy Forwarder), then the events are individual and easy to read. Like this:

Event 1:

 

Invalid user testfrom 111.222.333.444 port1111

 

Event 2:

 

pam_unix(sshd:auth):check pass; userunknown

 

Event 3:

 

pam_unix(sshd:auth):authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.222.333.444

 

Event 4:

 

Failed password forinvalid user testfrom 111.222.333.444 port1111 ssh2

 

Event 5:

 

error: Received disconnectfrom 111.222.333.444 port1111:13: Unable toauthenticate [preauth]

 

Event 6:

 

Disconnected from invaliduser test 111.222.333.444port 1111 [preauth]

 

--------------------------------

I'm looking for a solution that I can send the journald data as shown in the figure above, but the journald data will be sent as in the second case.

Thanks in advance for your help.

Labels (4)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Unfortunately, as you're introducing an additional external component (cribl worker), it's hard to say what happens where.

BTW, it's probably not that cribl merges anything, more like it doesn't split the events properly since UF sends data in chunks, not single events.

So the one at fault here is most probably the cribl one. BTW, why don't you just send UF->Indexer (or UF->HF->Indexer)?

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...