Getting Data In

Why are events using the wrong 'host' value?

rjk123
Explorer

I have configured a Splunk HF with the following inputs.conf stanzas (details changed) for two new device logs. Note the explicit host setting for each:

[monitor:///path/splunklogs/10.10.1.1/*.log]
disabled = false
host = myhost10
sourcetype = syslog
index = my_index

[monitor:///path/splunklogs/10.20.1.1/*.log]
disabled = false
host = myhost20
sourcetype = syslog
index = my_index

I created the index at the same time, so to validate its working I simply ran a search "index=my_index" (knowing there will be nothing else there). But surprisingly, the search returns events from three hosts instead of two!

The first device looks okay, but for the second one (ie. the second inputs stanza), some of the events are showing the wrong host value. It seems to be picking up a host value embedded in the event, but I don't see how. And I thought the inputs 'host' setting would override that anyway.

So, from the below example, the host SHOULD be set to 'myhost20' from the inputs stanza, but instead is showing as host 'xyz000000001234'.

Can anyone explain how that could be happening, and so, how to prevent it?

Sample event, with the standard fields below it:

2023-06-08T14:38:51+10:00 Sev=notice Facility=user Hostname=<loadbalancer> Header="Client " Message="Client IP: 10.20.1.1 | <109>Jun 8 14:40:11 xyz000000001234 some_field -: AUDIT [dvc="10.20.1.1" dvchost="10.20.1.1" version="7.7" user="<user>" role="" source="10.1.2.3" type="user_action" outcome="success" message="2023-06-08T14:40:11+10:00 abc120000001111 sshd\[2876138\]: Accepted keyboard-interactive/pam for device from 10.1.2.3 port 12345 ssh2"]"

host = xyz000000001234
index = my_index
source = /path/splunklogs/10.20.1.1/10.20.1.1-08-06-2023:14.log
sourcetype = syslog

 

Thanks for any response.

R.

Labels (1)
0 Karma
1 Solution

rjk123
Explorer

It's been a while, but I've found the problem.

The syslog traffic is coming through a load balancer, so hitting different HFs - needless to say, the HFs are not configured the same.

The problematic HF is newer and does not have the custom settings on the older one.

The new one is using the default props.conf file, and the [syslog] sourcetype stanza points to a [syslog-host] transforms stanza. The transform as a regex extraction to pick out the hostname - and that is overriding any host setting in the inputs.conf file.

The older HF has a custom [syslog] stanza which is using a different transform. Hence no host extraction, keeping the correct value intact.

The default transform causing the problem..

../etc/system/default/transforms.conf
[syslog-host]
DEST_KEY = MetaData:Host
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
FORMAT = host::$1

 

The end.

 

 

View solution in original post

0 Karma

rjk123
Explorer

The event detail I included is as it appears in the splunk search, only the host/path/index/user details are changed.

Yes, there are many other inputs on the HF, but only these for the new index.

What I mostly don't understand is why the 'host' field setting doesn't override whatever Splunk is doing to pickup the host value. Is it a not-so-powerful setting?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @rjk123,

it's really difficoult to debug your problem without seeing the events!
Anyway, the events with the wrong host are the same or different tha the others?

have you other inputs in that HF?

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @rjk123,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

rjk123
Explorer

It's been a while, but I've found the problem.

The syslog traffic is coming through a load balancer, so hitting different HFs - needless to say, the HFs are not configured the same.

The problematic HF is newer and does not have the custom settings on the older one.

The new one is using the default props.conf file, and the [syslog] sourcetype stanza points to a [syslog-host] transforms stanza. The transform as a regex extraction to pick out the hostname - and that is overriding any host setting in the inputs.conf file.

The older HF has a custom [syslog] stanza which is using a different transform. Hence no host extraction, keeping the correct value intact.

The default transform causing the problem..

../etc/system/default/transforms.conf
[syslog-host]
DEST_KEY = MetaData:Host
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(\w[\w\.\-]{2,})\]?\s
FORMAT = host::$1

 

The end.

 

 

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...