Get the journald logs from Universal Forwarder in JSON format to Splunk

Uzumaki — Tue, 19 Mar 2024 07:46:06 GMT

Hello,

first of all, sorry for my bad English, I hope you can understand everything.

My goal is to get the journald logs from the universalforwarder in JSON format to Splunk. (Splunk/UF Version 9.1.2)

I use the app jorunald_input.

inputs.conf (UF)

[journald://sshd] index = test sourcetype = test journalctl-filter = _SYSTEMD_UNIT=sshd.service

I've tried different props.conf functions. For example, something like this:

props.conf (UF)

[test] INDEXED_EXTRACTIONS = json KV_MODE = json SHOULD_LINEMERGE=false #INDEXED_EXTRACTIONS =json #NO_BINARY_CHECK=true #AUTO_KV_JSON = true

On the UF I check with the command

ps aux | grep journalctl

whether the query is enabled.
It displays this command

journalctl -f -o json --after-cursor s=a12345ab1abc12ab12345a01f1e920538;i=43a2c;b=c7efb124c33f43b0b0142ca0901ca8de;m=11aa0e450a21;t=233ae3422cd31;x=00af2c733a2cdfe7 _SYSTEMD_UNIT=sshd.service -q --output-fields PRIORITY,_SYSTEMD_UNIT,_SYSTEMD_CGROUP,_TRANSPORT,_PID,_UID,_MACHINE_ID,_GID,_COMM,_EXE,MESSAGE

I can try it out by using this command in the cli

But I have to take out that part "--after-cursor ...."

So I run the following command on the CLI to keep track of the journald logs:

journalctl -f -o json _SYSTEMD_UNIT=sshd.service -q --output-fields PRIORITY,_SYSTEMD_UNIT,_SYSTEMD_CGROUP,_TRANSPORT,_PID,_UID,_MACHINE_ID,_GID,_COMM,_EXE,MESSAGE

On the Universal forwarder, the tracked journald logs will then look like this:

(It would be a nice JSON format)

{ "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a2c;b=a1aaa111a11aaa111aa000a0101;m=11aa00c5b9a0;t=233ae39a37aa2;x=00af2c733a2cdfe7", "__REALTIME_TIMESTAMP" : "1710831664593570", "__MONOTONIC_TIMESTAMP" : "27194940570016", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "PRIORITY" : "6", "_UID" : "0", "_MACHINE_ID" : "1111", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "MESSAGE" : "Invalid user asdf from 111.11.111.111 port 111", "_PID" : "1430615" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a2d;b=a1aaa111a11aaa111aa000a0101;m=11aa00ec25bf;t=233ae39c9e6c0;x=10ac2c735c2cdfe7", "__REALTIME_TIMESTAMP" : "1710831667111616", "__MONOTONIC_TIMESTAMP" : "27194943088063", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "_UID" : "0", "_MACHINE_ID" : "1111", "PRIORITY" : "5", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "MESSAGE" : "pam_unix(sshd:auth): check pass; user unknown", "_PID" : "1430615" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a2e;b=a1aaa111a11aaa111aa000a0101;m=11aa00ec278a;t=233ae39c9e88c;x=5fb4c21ae6130519", "__REALTIME_TIMESTAMP" : "1710831667112076", "__MONOTONIC_TIMESTAMP" : "27194943088522", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "_UID" : "0", "_MACHINE_ID" : "1111", "PRIORITY" : "5", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "MESSAGE" : "pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.11.111.111", "_PID" : "1430615" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a2f;b=a1aaa111a11aaa111aa000a0101;m=11aa0108f5bf;t=233ae39e6b6c0;x=d072e90acf887129", "__REALTIME_TIMESTAMP" : "1710831668999872", "__MONOTONIC_TIMESTAMP" : "27194944976319", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "PRIORITY" : "6", "_UID" : "0", "_MACHINE_ID" : "1111", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "_PID" : "1430615", "MESSAGE" : "Failed password for invalid user asdf from 111.11.111.111 port 111 ssh2" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a30;b=a1aaa111a11aaa111aa000a0101;m=11aa010e0295;t=233ae39ebc397;x=d1eb29e00003daa7", "__REALTIME_TIMESTAMP" : "1710831669330839", "__MONOTONIC_TIMESTAMP" : "27194945307285", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "_UID" : "0", "_MACHINE_ID" : "1111", "PRIORITY" : "5", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "MESSAGE" : "pam_unix(sshd:auth): check pass; user unknown", "_PID" : "1430615" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a31;b=a1aaa111a11aaa111aa000a0101;m=11aa012f0b3c;t=233ae3a0ccc3e;x=c33e28a6111c89ea", "__REALTIME_TIMESTAMP" : "1710831671495742", "__MONOTONIC_TIMESTAMP" : "27194947472188", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "PRIORITY" : "6", "_UID" : "0", "_MACHINE_ID" : "1111", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "_PID" : "1430615", "MESSAGE" : "Failed password for invalid user asdf from 111.11.111.111 port 111 ssh2" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a32;b=a1aaa111a11aaa111aa000a0101;m=11aa0135591b;t=233ae3a131a1d;x=45420f6d2ca07377", "__REALTIME_TIMESTAMP" : "1710831671908893", "__MONOTONIC_TIMESTAMP" : "27194947885339", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "_UID" : "0", "_MACHINE_ID" : "1111", "_GID" : "0", "PRIORITY" : "3", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "_PID" : "1430615", "MESSAGE" : "error: Received disconnect from 111.11.111.111 port 111:11: Unable to authenticate [preauth]" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a33;b=a1aaa111a11aaa111aa000a0101;m=11aa01355bee;t=233ae3a131cf0;x=15b1aa1201a45cdf", "__REALTIME_TIMESTAMP" : "1710831671909616", "__MONOTONIC_TIMESTAMP" : "27194947886062", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "PRIORITY" : "6", "_UID" : "0", "_MACHINE_ID" : "1111", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "_PID" : "1430615", "MESSAGE" : "Disconnected from invalid user asdf 111.11.111.111 port 111 [preauth]" } { "__CURSOR" : "s=a12345ab1abc12ab12345a01f1e920538;i=43a34;b=a1aaa111a11aaa111aa000a0101;m=11aa01355c42;t=233ae3a131d45;x=123f45a09e00a8a2", "__REALTIME_TIMESTAMP" : "1710831671909701", "__MONOTONIC_TIMESTAMP" : "27194947886146", "_BOOT_ID" : "a1aaa111a11aaa111aa000a0101", "_TRANSPORT" : "syslog", "_UID" : "0", "_MACHINE_ID" : "1111", "PRIORITY" : "5", "_GID" : "0", "_COMM" : "sshd", "_EXE" : "/usr/sbin/sshd", "_SYSTEMD_CGROUP" : "/system.slice/sshd.service", "_SYSTEMD_UNIT" : "sshd.service", "MESSAGE" : "PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.11.111.111", "_PID" : "1430615" }

(Example)

But when I look for the logs on the search head, they look like this:

Invalid user asdf from 111.11.111.111 port 111pam_unix(sshd:auth): check pass; user unknownpam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.11.111.111Failed password for invalid user asdf from 111.11.111.111 port 111 ssh2pam_unix(sshd:auth): check pass; user unknownFailed password for invalid user asdf from 111.11.111.111 port 111 ssh2error: Received disconnect from 111.11.111.111 port 111:11: Unable to authenticate [preauth]Disconnected from invalid user asdf 111.11.111.111 port 111 [preauth]PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.11.111.111

Does anyone know why the logs are written together and not to be considered individually?

And why the logs are not in JSON format?

Can anyone tell me a solution for this on how to fix the problem?

Thank you very much!

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

KendallW — Tue, 26 Mar 2024 00:17:38 GMT

Are there any sourcetype parsing issues in the splunkd.log on the receiving indexer/forwarder?

index=_internal host=<receiving indexer/forwarder> log_level!=INFO "test"

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Uzumaki — Wed, 10 Apr 2024 06:28:24 GMT

I can't see any sourcetype parsing issues.

I only see old bugs from testing the app. But these should not be necessary for us.

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Uzumaki — Wed, 10 Apr 2024 07:48:42 GMT

Or is there an option to tell Splunk to insert a separator between the events and not write them directly together?

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Uzumaki — Thu, 25 Apr 2024 10:01:41 GMT

is it important to know that I am sending the data via cribl workers and not using heavy forwarders?

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

KendallW — Tue, 04 Jun 2024 03:41:40 GMT

You mentioned in your post you are using UF to send the data. Is the data going from Splunk UF --> Cribl --> Splunk indexer?

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Uzumaki — Tue, 04 Jun 2024 04:34:06 GMT

yes, the data is sent from the Splunk UF --> Cribl (Stream / Worker) --> Splunk Indexer

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

KendallW — Tue, 04 Jun 2024 05:07:39 GMT

I do not know much about Cribl, but these settings in props.conf might help:

props.conf on UF:

[test] EVENT_BREAKER_ENABLE=true EVENT_BREAKER=([\r\n]+)\{ \"__CURSOR\"

props.conf on Indexer: (assuming REALTIME_TIMESTAMP is the timestamp field)

[test] KV_MODE=JSON SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+)\{ \"__CURSOR\" MUST_BREAK_AFTER=\} TIME_PREFIX=\"__REALTIME_TIMESTAMP\"\s\:\s\" TIME_FORMAT=%s%6N MAX_TIMESTAMP_LOOKAHEAD=18

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

PickleRick — Tue, 04 Jun 2024 07:39:11 GMT

It doesn't work like that. For TA_auditd to work you ingest contents of /var/log/audit/auditd.log in text form.

The settings you're trying to manipulate do completely different things - they tell Splunk how to _interpret_ the received data. You can't use them to make json from plain text or something like that.

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Uzumaki — Tue, 04 Jun 2024 08:24:49 GMT

Sorry, I've opened a new post about my problem. I think that I have given some wrong information here, which I have noticed in the meantime.

https://community.splunk.com/t5/Getting-Data-In/Collect-journalctl-events-with-a-Splunk-UF-to-Cribl-Stream-in/m-p/689510#M114765

topic Re: Get the journald logs from Universal Forwarder in JSON format to Splunk in Getting Data In

Get the journald logs from Universal Forwarder in JSON format to Splunk

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk

Re: Get the journald logs from Universal Forwarder in JSON format to Splunk