Splunk Enterprise

Why are all monitors working except /var/log/messages?

RoyceTheBiker
Explorer

I copied the default inputs.conf to local and added some monitor configurations. There are seven monitors setup but only six are reporting. /var/log/messages is not working.

[monitor:///var/log/messages]
disabled = false
#index = linuxlog
sourcetype = syslog

[monitor:///var/log/secure]
disabled = 0
sourcetype = linux_secure

The logs from secure do show up and the other monitors are working as expected.

I have tried with ``disabled = 0``, also with and without index and sourcetype. 

All the examples I am finding indicate that this should be working. 

Labels (1)
Tags (2)
0 Karma
1 Solution

RoyceTheBiker
Explorer

I found that the Splunk configurations for syslog get the hostname from the log file itself. 

I set /etc/rsyslog.conf to have $PreserveFQDN on and now it is working correctly.

View solution in original post

0 Karma

ephemeric
Contributor

You could also set the host here:

[monitor:///var/log/messages]
disabled = false
host = <FQDN>
sourcetype = syslog
0 Karma

RoyceTheBiker
Explorer

I just found that the /var/log/messages are being sent to the Splunk server, but the hostname is the short name, not the FQDN that I was looking at.

Under the host=<short name> the only source is /var/log/message.  

I will look to see if there is another spot where the short name is used and forces this.

0 Karma

RoyceTheBiker
Explorer

I found that the Splunk configurations for syslog get the hostname from the log file itself. 

I set /etc/rsyslog.conf to have $PreserveFQDN on and now it is working correctly.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

When your UF runs as non root user you must give access rights to needed log files an directories. If you are running also selinux  then you must also modify its policy.

r. Ismo

0 Karma

RoyceTheBiker
Explorer

Both messages and secure are root:root 600.  The UF (splunkd) is being run by root so is also the process-runner.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Are you running selinux? Did you find any relevant error from messages or auditd files?

0 Karma

RoyceTheBiker
Explorer

SE is disabled. 
This looks like there are no errors with the config files.

/var/log/messages
Jul 24 20:20:43 devtools splunk: All installed files intact.
Jul 24 20:20:43 devtools splunk: Done
Jul 24 20:20:43 devtools splunk: All preliminary checks passed.
Jul 24 20:20:43 devtools splunk: Starting splunk server daemon (splunkd)...
Jul 24 20:20:43 devtools splunk: Done

0 Karma

RoyceTheBiker
Explorer

The only splunk lines in the audit.log is for when systemd restarts the service.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...