Why are all monitors working except /var/log/messages?

[RoyceTheBiker]

I copied the default inputs.conf to local and added some monitor configurations. There are seven monitors setup but only six are reporting. /var/log/messages is not working.

[monitor:///var/log/messages]
disabled = false
#index = linuxlog
sourcetype = syslog

[monitor:///var/log/secure]
disabled = 0
sourcetype = linux_secure

The logs from secure do show up and the other monitors are working as expected.

I have tried with ``disabled = 0``, also with and without index and sourcetype. 

All the examples I am finding indicate that this should be working. 

[ephemeric]

You could also set the host here:

[monitor:///var/log/messages]
disabled = false
host = <FQDN>
sourcetype = syslog

[RoyceTheBiker]

I just found that the /var/log/messages are being sent to the Splunk server, but the hostname is the short name, not the FQDN that I was looking at.

Under the host=<short name> the only source is /var/log/message.  

I will look to see if there is another spot where the short name is used and forces this.

[RoyceTheBiker · Accepted solution]

I found that the Splunk configurations for syslog get the hostname from the log file itself. 

I set /etc/rsyslog.conf to have $PreserveFQDN on and now it is working correctly.

[isoutamo]

When your UF runs as non root user you must give access rights to needed log files an directories. If you are running also selinux  then you must also modify its policy.

r. Ismo

[RoyceTheBiker]

Both messages and secure are root:root 600.  The UF (splunkd) is being run by root so is also the process-runner.

[isoutamo]

Are you running selinux? Did you find any relevant error from messages or auditd files?

[RoyceTheBiker]

SE is disabled. 
This looks like there are no errors with the config files.

/var/log/messages
Jul 24 20:20:43 devtools splunk: All installed files intact.
Jul 24 20:20:43 devtools splunk: Done
Jul 24 20:20:43 devtools splunk: All preliminary checks passed.
Jul 24 20:20:43 devtools splunk: Starting splunk server daemon (splunkd)...
Jul 24 20:20:43 devtools splunk: Done

[RoyceTheBiker]

The only splunk lines in the audit.log is for when systemd restarts the service.