unconfigured host showing up in results

[gurinderbhatti]

when searching for a specific index and sourcetype, the results come from a host that is not configured anywhere in serverclass.conf

i configured an inputs.conf file which monitors "/var/syslog" and "/x/y/z/WebApp/WebApp.log"

this config was pushed out to 2 hosts hostA and hostB

when i do the below search
index=lnx_appservers source="/x/y/z/WebApp/WebApp.log"
the results show up as coming from hostC (not configured anywhere on my deployment server)

but if i do search for index=lnx_appservers host=hostA OR host=hostB
the source in the results is /var/syslog

So bascially , where am i getting hostC from? why does /var/syslog show up but not the webapp.log when i search for hostA or hostB in that index?

[martin_mueller]

So... hostC is expected to run a UF? Check if it has any of the webapp apps in its etc/apps directory.

[gurinderbhatti]

thx Martin,
there are no transforms.conf for this source and index.
this hostC is showing up in another index (lnx_splunk)
the conf file monitors multiple statistical parameters
[monitor:///root/.bash_history]
index=lnx_splunk

[monitor:///home/.../.bash_history]
index=lnx_splunk

[script://./bin/openPortsEnhanced.sh]
index=lnx_splunk

[script://./bin/service.sh]
index=lnx_splunk

[script://./bin/sshdChecker.sh]
but my question remains, this hos is never mentioned in serverclass.conf, so what config is getting pushed to it and why is it associating with the lnx_appservers index

[martin_mueller]

Do you see a third forwarder host in _internal?
Are there any transforms.conf rules set for sourcetype app_webapp or source /x/y/z/WebApp/WebApp.log or any of your hosts?

[yannK]

Remember that for the sourcetype "syslog", the host is extracted from the event at index time.
Can you check your logs for hostC, and see if the host is not mentioned in it.
Maybe you have a syslog collector, that is receiving logs from remote servers and write them to the default /var/log/message

[gurinderbhatti]

well the sourcetype syslog doesnt show up for hostC , and it should not, it only and correctly shows up for hostA and hostB.
But why does sourcetype /webapp.log show up for hostC. its not configured anywhere in serverclass yet it shows up for my lnx_appservers index as well as another index lnx_splunk (for system releated events i.e. iostat, vmstats,ps,etc)
any ideas?

[gurinderbhatti]

Thanks Martin, below is my inputs.conf file:

System logs

[monitor:///var/adm]
index=lnx_appservers
whitelist=(.log|log$|messages)
disabled = 0

Application Logs

[monitor:///x/y/z/WebApp/WebApp.log]
index = lnx_appservers
sourcetype = app_webapp
disabled = false
ignoreOlderThan = 7d
my serverclass looks like this:
[serverClass:lnx_webapp]
whitelist.0 = hostA*
whitelist.1 = hostB*
restartSplunkd = true
[serverClass:lnx_webapp:app:deploymentclient]
[serverClass:lnx_webapp:app:lnx_webapp_inputs]
[serverClass:lnx_webapp:app:lnx_webapp_props]
[serverClass:lnx_webapp:app:forwarder_outputs]

[martin_mueller]

Do note, a forwarder installed on hostA is perfectly capable of producing events with Splunk's host field set to hostC. Simple examples are when you set the host field in the inputs.conf stanza, more complex examples extract the host from the source data - quite common in syslog data.

Additionally, there may be forwarders sending data that aren't configured in your deployment server. Check the _internal index for that.

[donald_mccarthy]

Is there a CNAME record for either hostA or hostB?