Getting Data In
Highlighted

unconfigured host showing up in results

Path Finder

when searching for a specific index and sourcetype, the results come from a host that is not configured anywhere in serverclass.conf

i configured an inputs.conf file which monitors "/var/syslog" and "/x/y/z/WebApp/WebApp.log"

this config was pushed out to 2 hosts hostA and hostB

when i do the below search
index=lnx_appservers source="/x/y/z/WebApp/WebApp.log"
the results show up as coming from hostC (not configured anywhere on my deployment server)

but if i do search for index=lnx_appservers host=hostA OR host=hostB
the source in the results is /var/syslog

So bascially , where am i getting hostC from? why does /var/syslog show up but not the webapp.log when i search for hostA or hostB in that index?

0 Karma
Highlighted

Re: unconfigured host showing up in results

Is there a CNAME record for either hostA or hostB?

0 Karma
Highlighted

Re: unconfigured host showing up in results

SplunkTrust
SplunkTrust

Do note, a forwarder installed on hostA is perfectly capable of producing events with Splunk's host field set to hostC. Simple examples are when you set the host field in the inputs.conf stanza, more complex examples extract the host from the source data - quite common in syslog data.

Additionally, there may be forwarders sending data that aren't configured in your deployment server. Check the _internal index for that.

0 Karma
Highlighted

Re: unconfigured host showing up in results

Path Finder

Thanks Martin, below is my inputs.conf file:

System logs

[monitor:///var/adm]
index=lnx_appservers
whitelist=(.log|log$|messages)
disabled = 0

Application Logs

[monitor:///x/y/z/WebApp/WebApp.log]
index = lnxappservers
sourcetype = app
webapp
disabled = false
ignoreOlderThan = 7d
my serverclass looks like this:
[serverClass:lnx_webapp]
whitelist.0 = hostA*
whitelist.1 = hostB*
restartSplunkd = true
[serverClass:lnxwebapp:app:deploymentclient]
[serverClass:lnxwebapp:app:lnxwebappinputs]
[serverClass:lnx
webapp:app:lnxwebappprops]
[serverClass:lnx
webapp:app:forwarder_outputs]

0 Karma
Highlighted

Re: unconfigured host showing up in results

Splunk Employee
Splunk Employee

Remember that for the sourcetype "syslog", the host is extracted from the event at index time.
Can you check your logs for hostC, and see if the host is not mentioned in it.
Maybe you have a syslog collector, that is receiving logs from remote servers and write them to the default /var/log/message

0 Karma
Highlighted

Re: unconfigured host showing up in results

Path Finder

well the sourcetype syslog doesnt show up for hostC , and it should not, it only and correctly shows up for hostA and hostB.
But why does sourcetype /webapp.log show up for hostC. its not configured anywhere in serverclass yet it shows up for my lnxappservers index as well as another index lnxsplunk (for system releated events i.e. iostat, vmstats,ps,etc)
any ideas?

0 Karma
Highlighted

Re: unconfigured host showing up in results

SplunkTrust
SplunkTrust

Do you see a third forwarder host in _internal?
Are there any transforms.conf rules set for sourcetype app_webapp or source /x/y/z/WebApp/WebApp.log or any of your hosts?

0 Karma
Highlighted

Re: unconfigured host showing up in results

Path Finder

thx Martin,
there are no transforms.conf for this source and index.
this hostC is showing up in another index (lnxsplunk)
the conf file monitors multiple statistical parameters
[monitor:///root/.bash
history]
index=lnx_splunk

[monitor:///home/.../.bashhistory]
index=lnx
splunk

[script://./bin/openPortsEnhanced.sh]
index=lnx_splunk

[script://./bin/service.sh]
index=lnx_splunk

[script://./bin/sshdChecker.sh]
but my question remains, this hos is never mentioned in serverclass.conf, so what config is getting pushed to it and why is it associating with the lnx_appservers index

0 Karma
Highlighted

Re: unconfigured host showing up in results

SplunkTrust
SplunkTrust

So... hostC is expected to run a UF? Check if it has any of the webapp apps in its etc/apps directory.

0 Karma