Why are events using the wrong 'host' value?

[rjk123]

I have configured a Splunk HF with the following inputs.conf stanzas (details changed) for two new device logs. Note the explicit host setting for each:

[monitor:///path/splunklogs/10.10.1.1/*.log]
disabled = false
host = myhost10
sourcetype = syslog
index = my_index

[monitor:///path/splunklogs/10.20.1.1/*.log]
disabled = false
host = myhost20
sourcetype = syslog
index = my_index

I created the index at the same time, so to validate its working I simply ran a search "index=my_index" (knowing there will be nothing else there). But surprisingly, the search returns events from three hosts instead of two!

The first device looks okay, but for the second one (ie. the second inputs stanza), some of the events are showing the wrong host value. It seems to be picking up a host value embedded in the event, but I don't see how. And I thought the inputs 'host' setting would override that anyway.

So, from the below example, the host SHOULD be set to 'myhost20' from the inputs stanza, but instead is showing as host 'xyz000000001234'.

Can anyone explain how that could be happening, and so, how to prevent it?

Sample event, with the standard fields below it:

2023-06-08T14:38:51+10:00 Sev=notice Facility=user Hostname=<loadbalancer> Header="Client " Message="Client IP: 10.20.1.1 | <109>Jun 8 14:40:11 xyz000000001234 some_field -: AUDIT [dvc="10.20.1.1" dvchost="10.20.1.1" version="7.7" user="<user>" role="" source="10.1.2.3" type="user_action" outcome="success" message="2023-06-08T14:40:11+10:00 abc120000001111 sshd\[2876138\]: Accepted keyboard-interactive/pam for device from 10.1.2.3 port 12345 ssh2"]"

host = xyz000000001234
index = my_index
source = /path/splunklogs/10.20.1.1/10.20.1.1-08-06-2023:14.log
sourcetype = syslog

Thanks for any response.

R.

[rjk123]

The event detail I included is as it appears in the splunk search, only the host/path/index/user details are changed.

Yes, there are many other inputs on the HF, but only these for the new index.

What I mostly don't understand is why the 'host' field setting doesn't override whatever Splunk is doing to pickup the host value. Is it a not-so-powerful setting?

[gcusello]

Hi @rjk123,

it's really difficoult to debug your problem without seeing the events!
Anyway, the events with the wrong host are the same or different tha the others?

have you other inputs in that HF?

Ciao.

Giuseppe

[gcusello]

Hi @rjk123,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[rjk123 · Accepted solution]

It's been a while, but I've found the problem.

The syslog traffic is coming through a load balancer, so hitting different HFs - needless to say, the HFs are not configured the same.

The problematic HF is newer and does not have the custom settings on the older one.

The new one is using the default props.conf file, and the [syslog] sourcetype stanza points to a [syslog-host] transforms stanza. The transform as a regex extraction to pick out the hostname - and that is overriding any host setting in the inputs.conf file.

The older HF has a custom [syslog] stanza which is using a different transform. Hence no host extraction, keeping the correct value intact.

The default transform causing the problem..

../etc/system/default/transforms.conf
[syslog-host]
DEST_KEY = MetaData:Host
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
FORMAT = host::$1

The end.