/var/log/messages associated with index host NOT the correct source host

[krussell101]

I have two dozen UF linux systems. All of them are picking up /var/log/messages and sending it to my indexer (the one and only "splunk" host).

All of the /var/log/messages entries are indexed as coming from host=splunk.

WHY?

HELP!

Thanks.

[BryanBerry · Accepted solution]

Have any stray props.conf/transforms.conf on the indexer? Is this all of your forwarders or only some of them?

Try "splunk btool transforms list --debug > out.txt" on your indexer and grep for MetaData:Host in out.txt. It possible that there's a transform setting the host value to splunk

[BryanBerry]

Huh, that is very odd. Was that in etc/system/local/transforms.conf or default? Did you find how that got in there? I'm curious about the cause as well.

[krussell101]

FOUND IT! THANK YOU!

[krussell101]

Any hints on where this may have snuck in? I don't recall doing any transforms at all. Will this be on my forwarders somewhere or on the indexer?

Thanks again.

[krussell101]

This is occurring to /var/log/messages from all of my forwarders.

transforms list output:

system [syslog-host]
system CAN_OPTIMIZE = True
system CLEAN_KEYS = True
system DEFAULT_VALUE =
system DEST_KEY = MetaData:Host
system FORMAT = host::splunk-mydomain.com

Is this taking everything of sourcetype syslog and attributing it to another host?

Cool! So now I have to unset that somehow. Thanks!

[krussell101]

This isn't an answer

[krussell101]

No. they pick up the local hostname. Which is how I want it to behave. All other files in /var/log come over with the correct hostname. This is the only file which is attributed to the incorrect host.

[Lucas_K]

Is there a hostname defined on your UF's in /opt/splunkforwarder/etc/system/local/inputs.conf?

[krussell101]

sourcetype = syslog

[Ayn]

What sourcetype do you have for /var/log/messages?