topic Re: All monitors are working except /var/log/messages. in Splunk Enterprise

Why are all monitors working except /var/log/messages?

RoyceTheBiker — Fri, 24 Jul 2020 22:29:20 GMT

I copied the default inputs.conf to local and added some monitor configurations. There are seven monitors setup but only six are reporting. /var/log/messages is not working.

[monitor:///var/log/messages]
disabled = false
#index = linuxlog
sourcetype = syslog

[monitor:///var/log/secure]
disabled = 0
sourcetype = linux_secure

The logs from secure do show up and the other monitors are working as expected.

I have tried with ``disabled = 0``, also with and without index and sourcetype.

All the examples I am finding indicate that this should be working.

Re: All monitors are working except /var/log/messages.

isoutamo — Fri, 24 Jul 2020 20:16:23 GMT

When your UF runs as non root user you must give access rights to needed log files an directories. If you are running also selinux then you must also modify its policy.

r. Ismo

Re: All monitors are working except /var/log/messages.

RoyceTheBiker — Fri, 24 Jul 2020 20:47:21 GMT

Both messages and secure are root:root 600. The UF (splunkd) is being run by root so is also the process-runner.

Re: All monitors are working except /var/log/messages.

isoutamo — Fri, 24 Jul 2020 20:55:15 GMT

Are you running selinux? Did you find any relevant error from messages or auditd files?

Re: All monitors are working except /var/log/messages.

RoyceTheBiker — Fri, 24 Jul 2020 21:04:13 GMT

SE is disabled.
This looks like there are no errors with the config files.

/var/log/messages
Jul 24 20:20:43 devtools splunk: All installed files intact.
Jul 24 20:20:43 devtools splunk: Done
Jul 24 20:20:43 devtools splunk: All preliminary checks passed.
Jul 24 20:20:43 devtools splunk: Starting splunk server daemon (splunkd)...
Jul 24 20:20:43 devtools splunk: Done

Re: All monitors are working except /var/log/messages.

RoyceTheBiker — Fri, 24 Jul 2020 21:08:43 GMT

The only splunk lines in the audit.log is for when systemd restarts the service.

Re: All monitors are working except /var/log/messages.

RoyceTheBiker — Fri, 24 Jul 2020 22:14:02 GMT

I just found that the /var/log/messages are being sent to the Splunk server, but the hostname is the short name, not the FQDN that I was looking at.

Under the host=<short name> the only source is /var/log/message.

I will look to see if there is another spot where the short name is used and forces this.

Re: All monitors are working except /var/log/messages.

RoyceTheBiker — Mon, 27 Jul 2020 15:32:36 GMT

I found that the Splunk configurations for syslog get the hostname from the log file itself.

I set /etc/rsyslog.conf to have $PreserveFQDN on and now it is working correctly.

Re: Why are all monitors working except /var/log/messages?

ephemeric — Mon, 23 Aug 2021 15:38:55 GMT

You could also set the host here:

[monitor:///var/log/messages] disabled = false host = <FQDN> sourcetype = syslog