We're updating our Linux Servers to Debian 12. A few host went "missing" afterwards in Splunk.
While investigating into it I found out that they were in fact not missing, but they stopped writing logfiles to /var/log. Seems like Debian switched to full journald, as I was promoted with this ReadMe in /var/log:
You are looking for the traditional text log files in /var/log, and they are
gone?
Here's an explanation on what's going on:
You are running a systemd-based OS where traditional syslog has been replaced
with the Journal. The journal stores the same (and more) information as classic
syslog. To make use of the journal and access the collected log data simply
invoke "journalctl", which will output the logs in the identical text-based
format the syslog files in /var/log used to be. For further details, please
refer to journalctl(1).
[...]
Of course we can simply install the rsyslog package again, but this is feels more like a step backwards.
So here is my question:
Is there a default and generic approach for collecting all system and service logs from journald that we can use on our UFs, since Logfiles are obviously not the future on Linux.
Best regards
Splunk supports journald inputs. See https://docs.splunk.com/Documentation/Splunk/9.1.0/Data/CollecteventsfromJournalD
Thanks for your answer!
Sorry for being unclear about the intention of my question. I know about the journald inputs but the stanza configs are kind of restricted in terms of filtering and therefore not very useful.
For Example, if I want to pick up sshd Logs form journald with an separate stanza I cloud do something like :
[journald://sshd]
journalctl-filter = _SYSTEMD_UNIT=sshd.service
This would ship all Logs of the sshd daemon and it's easy to search them by source="journald://sshd". I could repeat this for all the other services of interest.
However I don't want to miss all the other logs that are not covered by separate stanzas. Therefore I'll need a "catch all" stanza. But this will lead to a lot of duplicates based on the fact, that journald input has no good solution to filter already covered inputs. E.g: `journalctl-exclude-field = _SYSTEMD_UNIT=sshd.service` in my "catch-all"-stanza won't work.
If Journald is the future of logging on Linux, this can't be the solution Splunk is offering. Additionally, it's going to make source-typing useless because all inputs will end up with `sourcetype=journald` on the indexer.