Getting Data In

/var/log/messages associated with index host NOT the correct source host

krussell101
Path Finder

I have two dozen UF linux systems. All of them are picking up /var/log/messages and sending it to my indexer (the one and only "splunk" host).

All of the /var/log/messages entries are indexed as coming from host=splunk.

WHY?

HELP!

Thanks.

Tags (1)
0 Karma
1 Solution

BryanBerry
Path Finder

Have any stray props.conf/transforms.conf on the indexer? Is this all of your forwarders or only some of them?

Try "splunk btool transforms list --debug > out.txt" on your indexer and grep for MetaData:Host in out.txt. It possible that there's a transform setting the host value to splunk

View solution in original post

0 Karma

BryanBerry
Path Finder

Have any stray props.conf/transforms.conf on the indexer? Is this all of your forwarders or only some of them?

Try "splunk btool transforms list --debug > out.txt" on your indexer and grep for MetaData:Host in out.txt. It possible that there's a transform setting the host value to splunk

0 Karma

BryanBerry
Path Finder

Huh, that is very odd. Was that in etc/system/local/transforms.conf or default? Did you find how that got in there? I'm curious about the cause as well.

0 Karma

krussell101
Path Finder

FOUND IT! THANK YOU!

0 Karma

krussell101
Path Finder

Any hints on where this may have snuck in? I don't recall doing any transforms at all. Will this be on my forwarders somewhere or on the indexer?

Thanks again.

0 Karma

krussell101
Path Finder

This is occurring to /var/log/messages from all of my forwarders.

transforms list output:

system [syslog-host]
system CAN_OPTIMIZE = True
system CLEAN_KEYS = True
system DEFAULT_VALUE =
system DEST_KEY = MetaData:Host
system FORMAT = host::splunk-mydomain.com

Is this taking everything of sourcetype syslog and attributing it to another host?

Cool! So now I have to unset that somehow. Thanks!

0 Karma

krussell101
Path Finder

This isn't an answer

0 Karma

krussell101
Path Finder

No. they pick up the local hostname. Which is how I want it to behave. All other files in /var/log come over with the correct hostname. This is the only file which is attributed to the incorrect host.

0 Karma

Lucas_K
Motivator

Is there a hostname defined on your UF's in /opt/splunkforwarder/etc/system/local/inputs.conf?

krussell101
Path Finder

sourcetype = syslog

0 Karma

Ayn
Legend

What sourcetype do you have for /var/log/messages?

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...