Getting Data In

unconfigured host showing up in results

gurinderbhatti
Path Finder

when searching for a specific index and sourcetype, the results come from a host that is not configured anywhere in serverclass.conf

i configured an inputs.conf file which monitors "/var/syslog" and "/x/y/z/WebApp/WebApp.log"

this config was pushed out to 2 hosts hostA and hostB

when i do the below search
index=lnx_appservers source="/x/y/z/WebApp/WebApp.log"
the results show up as coming from hostC (not configured anywhere on my deployment server)

but if i do search for index=lnx_appservers host=hostA OR host=hostB
the source in the results is /var/syslog

So bascially , where am i getting hostC from? why does /var/syslog show up but not the webapp.log when i search for hostA or hostB in that index?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

So... hostC is expected to run a UF? Check if it has any of the webapp apps in its etc/apps directory.

0 Karma

gurinderbhatti
Path Finder

thx Martin,
there are no transforms.conf for this source and index.
this hostC is showing up in another index (lnx_splunk)
the conf file monitors multiple statistical parameters
[monitor:///root/.bash_history]
index=lnx_splunk

[monitor:///home/.../.bash_history]
index=lnx_splunk

[script://./bin/openPortsEnhanced.sh]
index=lnx_splunk

[script://./bin/service.sh]
index=lnx_splunk

[script://./bin/sshdChecker.sh]
but my question remains, this hos is never mentioned in serverclass.conf, so what config is getting pushed to it and why is it associating with the lnx_appservers index

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Do you see a third forwarder host in _internal?
Are there any transforms.conf rules set for sourcetype app_webapp or source /x/y/z/WebApp/WebApp.log or any of your hosts?

0 Karma

yannK
Splunk Employee
Splunk Employee

Remember that for the sourcetype "syslog", the host is extracted from the event at index time.
Can you check your logs for hostC, and see if the host is not mentioned in it.
Maybe you have a syslog collector, that is receiving logs from remote servers and write them to the default /var/log/message

0 Karma

gurinderbhatti
Path Finder

well the sourcetype syslog doesnt show up for hostC , and it should not, it only and correctly shows up for hostA and hostB.
But why does sourcetype /webapp.log show up for hostC. its not configured anywhere in serverclass yet it shows up for my lnx_appservers index as well as another index lnx_splunk (for system releated events i.e. iostat, vmstats,ps,etc)
any ideas?

0 Karma

gurinderbhatti
Path Finder

Thanks Martin, below is my inputs.conf file:

System logs

[monitor:///var/adm]
index=lnx_appservers
whitelist=(.log|log$|messages)
disabled = 0

Application Logs

[monitor:///x/y/z/WebApp/WebApp.log]
index = lnx_appservers
sourcetype = app_webapp
disabled = false
ignoreOlderThan = 7d
my serverclass looks like this:
[serverClass:lnx_webapp]
whitelist.0 = hostA*
whitelist.1 = hostB*
restartSplunkd = true
[serverClass:lnx_webapp:app:deploymentclient]
[serverClass:lnx_webapp:app:lnx_webapp_inputs]
[serverClass:lnx_webapp:app:lnx_webapp_props]
[serverClass:lnx_webapp:app:forwarder_outputs]

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Do note, a forwarder installed on hostA is perfectly capable of producing events with Splunk's host field set to hostC. Simple examples are when you set the host field in the inputs.conf stanza, more complex examples extract the host from the source data - quite common in syslog data.

Additionally, there may be forwarders sending data that aren't configured in your deployment server. Check the _internal index for that.

0 Karma

donald_mccarthy
Explorer

Is there a CNAME record for either hostA or hostB?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...