topic Re: Is there are default approach for journald syslog? in Getting Data In

Is there are default approach for journald syslog?

dsfyxcasdcertzu — Tue, 25 Jul 2023 11:00:42 GMT

We're updating our Linux Servers to Debian 12. A few host went "missing" afterwards in Splunk.

While investigating into it I found out that they were in fact not missing, but they stopped writing logfiles to /var/log. Seems like Debian switched to full journald, as I was promoted with this ReadMe in /var/log:

You are looking for the traditional text log files in /var/log, and they are
gone?

Here's an explanation on what's going on:

You are running a systemd-based OS where traditional syslog has been replaced
with the Journal. The journal stores the same (and more) information as classic
syslog. To make use of the journal and access the collected log data simply
invoke "journalctl", which will output the logs in the identical text-based
format the syslog files in /var/log used to be. For further details, please
refer to journalctl(1).
[...]

Of course we can simply install the rsyslog package again, but this is feels more like a step backwards.
So here is my question:

Is there a default and generic approach for collecting all system and service logs from journald that we can use on our UFs, since Logfiles are obviously not the future on Linux.

Best regards

Re: Is there are default approach for journald syslog?

richgalloway — Tue, 25 Jul 2023 12:04:51 GMT

Splunk supports journald inputs. See https://docs.splunk.com/Documentation/Splunk/9.1.0/Data/CollecteventsfromJournalD

Re: Is there are default approach for journald syslog?

dsfyxcasdcertzu — Wed, 26 Jul 2023 15:40:50 GMT

Thanks for your answer!

Sorry for being unclear about the intention of my question. I know about the journald inputs but the stanza configs are kind of restricted in terms of filtering and therefore not very useful.

For Example, if I want to pick up sshd Logs form journald with an separate stanza I cloud do something like :

[journald://sshd]
journalctl-filter = _SYSTEMD_UNIT=sshd.service

This would ship all Logs of the sshd daemon and it's easy to search them by source="journald://sshd". I could repeat this for all the other services of interest.

However I don't want to miss all the other logs that are not covered by separate stanzas. Therefore I'll need a "catch all" stanza. But this will lead to a lot of duplicates based on the fact, that journald input has no good solution to filter already covered inputs. E.g: `journalctl-exclude-field = _SYSTEMD_UNIT=sshd.service` in my "catch-all"-stanza won't work.

If Journald is the future of logging on Linux, this can't be the solution Splunk is offering. Additionally, it's going to make source-typing useless because all inputs will end up with `sourcetype=journald` on the indexer.

Re: Is there are default approach for journald syslog?

obpedro — Mon, 22 Dec 2025 23:04:43 GMT

Fast forward two years from your post, I'm dealing with the same issue. I'm thinking of just installing rsyslog and being done with it. It seems that Amazon themselves don't use journald for things like the cloudwatch agent, but instead recommend installing rsyslog and reading from the "traditional" log files.

Re: Is there are default approach for journald syslog?

PickleRick — Tue, 23 Dec 2025 09:22:48 GMT

Well, it's true that journald while being pushed on users isn't necessarily a logging tool of choice for many software authors so you can expect to have various pieces of software write directly to /var/log or send to local syslog daemon. You can indeed install rsyslog and have it read journald input. You can use journald in Splunk as well. And you don't have to specify single unit(s) in the input definition. If you don't filter, it should pick up all events.