All Apps and Add-ons

Splunk Add-on for Unix and Linux: Why are we seeing multiple hostnames for /var/log logfiles on Linux?

cmeerbeek
Path Finder

We have installed the Splunk Add-On for linux and enabled the /var/log monitor-stanza.
The data is stored in the correct syslog-sourcetype, but somehow the hostnames are mixed up.
/var/log/secure and /var/log/cron give us the FQDN but /var/log/message gives us the shortname for the same machine!
All logfiles have the shortname in every event, so this cannot be the issue.

Anyone seen this before and know how to solve this?

1 Solution

markbarber21
Path Finder

The "syslog-host" transform is applied, which overrides the host from the file output itself.

[syslog-host]
 DEST_KEY = MetaData:Host
 REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
 FORMAT = host::$1

See similar question here: https://answers.splunk.com/answers/43214/syslog-server-to-splunk-showing-incorrect-host-during-splun...

View solution in original post

markbarber21
Path Finder

The "syslog-host" transform is applied, which overrides the host from the file output itself.

[syslog-host]
 DEST_KEY = MetaData:Host
 REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
 FORMAT = host::$1

See similar question here: https://answers.splunk.com/answers/43214/syslog-server-to-splunk-showing-incorrect-host-during-splun...

cmeerbeek
Path Finder

The answer seems to be the fact that /var/log/messages is of sourcetype syslog and syslog gets a special treatment and one is the change of host field...
If someone can confirm this, case closed 🙂

0 Karma

acharlieh
Influencer

Built into Splunk for the syslog sourcetype is indeed a transformation to extract the host name from the log entry. The reason for this is often syslog from various devices is sent to a collection server before being sent on to Splunk (see http://www.georgestarcher.com/splunk-success-with-syslog/ ). Obviously the events you'd want to be labeled with the original device names and not the syslog server's hostname.

Edit to add... if you have access to the CLI on the Splunk server, you can use btool to check this out... first with $SPLUNK_HOME/bin/splunk btool props list syslog to see the line TRANSFORMS = syslog-host and then looking for that resolved transforms stanza as well.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...