How to find where logs are coming from and where they are being monitored?

[JoeSco27]

I am seeing logs in an instance of splunk, but i am unsure where the monitoring is set up. I checked my serverclass.conf and the servers were not listed on the whitelist. I checked my deployment monitor app and see 3 apps deployed to the server (my deploymentclient.conf app, my outputs.conf app, and windows app) when i check out each app there is no monitoring stanza for these logs I see in Splunk. I try to make a new serverClass but the logging that is already in place is taking priority and i cant format the logs.

Can someone help out with useful troubleshooting tricks or advice if they have seen this before?

[bmacias84]

If you are trying to figure out how which app contains the setting which are being set use btool.

./splunk cmd btool --debug inputs list
or
./splunk cmd btool --debug deploymentclient list

Also search your _internal index for downloads from your deployment server. The peer field should have ip address of the host in question with which serverclasses are being applied.

index=_internal PackageDownloadRestHandler

I am assuming you are sending your deployment server logs to your indexers and your are running 6.3 or higher.

[rlaan]

Not a splunk tool but if you are running on a *nix system, I usually run a command similar to this when trying to locate where files are coming from (sometimes transforms will rename sources so the inputs file wont contain the required information as to what is running the monitor, it is worth looking into transforms.conf if inputs.conf did not provide the source).

$ find /opt/splunk/etc -iname "*.conf" | xargs grep -Hni --color ""

this will search through all of the conf files under etc, return any lines to you the the search term was found in as well as displaying the path to the file it came from and the line number within that file. Easily my favorite command for searching systems I am unfamiliar with. Hope this help!

[Raghav2384]

usually source metafield will hold the location of the data source. index=|stats count by source should give you all the sources that are contributing to your splunk installation. Note : Assuming you have access to index= 🙂
Hope this helps.

Thanks,
Raghav

[JoeSco27]

I can see the source from which the log file is coming from, but as the sysAdmin I never set up monitoring for that source. I am trying to understand where in the config files this monitoring has been set up as I cannot see anything to do with it in my deployment-server's serverclass.conf or in my apps that include my inputs.conf.