Splunk Enterprise

Why are all monitors working except /var/log/messages?

RoyceTheBiker
Explorer

I copied the default inputs.conf to local and added some monitor configurations. There are seven monitors setup but only six are reporting. /var/log/messages is not working.

[monitor:///var/log/messages]
disabled = false
#index = linuxlog
sourcetype = syslog

[monitor:///var/log/secure]
disabled = 0
sourcetype = linux_secure

The logs from secure do show up and the other monitors are working as expected.

I have tried with ``disabled = 0``, also with and without index and sourcetype. 

All the examples I am finding indicate that this should be working. 

Labels (1)
Tags (2)
0 Karma
1 Solution

RoyceTheBiker
Explorer

I found that the Splunk configurations for syslog get the hostname from the log file itself. 

I set /etc/rsyslog.conf to have $PreserveFQDN on and now it is working correctly.

View solution in original post

0 Karma

ephemeric
Contributor

You could also set the host here:

[monitor:///var/log/messages]
disabled = false
host = <FQDN>
sourcetype = syslog
0 Karma

RoyceTheBiker
Explorer

I just found that the /var/log/messages are being sent to the Splunk server, but the hostname is the short name, not the FQDN that I was looking at.

Under the host=<short name> the only source is /var/log/message.  

I will look to see if there is another spot where the short name is used and forces this.

0 Karma

RoyceTheBiker
Explorer

I found that the Splunk configurations for syslog get the hostname from the log file itself. 

I set /etc/rsyslog.conf to have $PreserveFQDN on and now it is working correctly.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

When your UF runs as non root user you must give access rights to needed log files an directories. If you are running also selinux  then you must also modify its policy.

r. Ismo

0 Karma

RoyceTheBiker
Explorer

Both messages and secure are root:root 600.  The UF (splunkd) is being run by root so is also the process-runner.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Are you running selinux? Did you find any relevant error from messages or auditd files?

0 Karma

RoyceTheBiker
Explorer

SE is disabled. 
This looks like there are no errors with the config files.

/var/log/messages
Jul 24 20:20:43 devtools splunk: All installed files intact.
Jul 24 20:20:43 devtools splunk: Done
Jul 24 20:20:43 devtools splunk: All preliminary checks passed.
Jul 24 20:20:43 devtools splunk: Starting splunk server daemon (splunkd)...
Jul 24 20:20:43 devtools splunk: Done

0 Karma

RoyceTheBiker
Explorer

The only splunk lines in the audit.log is for when systemd restarts the service.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...