Splunk Add-on for Unix and Linux: Why are we seeing multiple hostnames for /var/log logfiles on Linux?

[cmeerbeek]

We have installed the Splunk Add-On for linux and enabled the /var/log monitor-stanza.
The data is stored in the correct syslog-sourcetype, but somehow the hostnames are mixed up.
/var/log/secure and /var/log/cron give us the FQDN but /var/log/message gives us the shortname for the same machine!
All logfiles have the shortname in every event, so this cannot be the issue.

Anyone seen this before and know how to solve this?

[markbarber21 · Accepted solution]

The "syslog-host" transform is applied, which overrides the host from the file output itself.

[syslog-host]
 DEST_KEY = MetaData:Host
 REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
 FORMAT = host::$1

See similar question here: https://answers.splunk.com/answers/43214/syslog-server-to-splunk-showing-incorrect-host-during-splun...

[cmeerbeek]

The answer seems to be the fact that /var/log/messages is of sourcetype syslog and syslog gets a special treatment and one is the change of host field...
If someone can confirm this, case closed 🙂

[acharlieh]

Built into Splunk for the syslog sourcetype is indeed a transformation to extract the host name from the log entry. The reason for this is often syslog from various devices is sent to a collection server before being sent on to Splunk (see http://www.georgestarcher.com/splunk-success-with-syslog/ ). Obviously the events you'd want to be labeled with the original device names and not the syslog server's hostname.

Edit to add... if you have access to the CLI on the Splunk server, you can use btool to check this out... first with $SPLUNK_HOME/bin/splunk btool props list syslog to see the line TRANSFORMS = syslog-host and then looking for that resolved transforms stanza as well.