Mysterious Log Feed

StuartMacL — Thu, 09 Nov 2023 11:31:41 GMT

I have a log feed which was configured by a previous employee. Documentation does not exist, of course...

The feed stopped once we migrated indexers. I checked the deployment server and there does not seem to be any apps on this server where the feed exists which ingest this feed.

Then, we added in the old indexer to the architecture and the feed started working again!

When I check these events in Search & Reporting, I can see the feed is only coming in via this legacy indexer by checking the splunk_server field.

I logged into the legacy indexer via CLI and used btool on inputs for the index, source and sourcetypes, no matches. Also struggling to find anything useful in the _internal events via search head GUI.

Both indexers are in a cluster and so the config should be identical, but the events only come in via the legacy indexer.

How can I find how this feed is configured?

Re: Mysterious Log Feed

richgalloway — Thu, 09 Nov 2023 15:35:17 GMT

One or more of the forwarders are either not using the Deployment Server, are not in a serverclass in the DS, or have outputs.conf set in $SPLUNK_HOME/etc/system/local (which overrides settings from the DS). The forwarder(s) still has the old indexer configured and is not getting the new indexer list from anywhere.

You'll have to sign in to the forwarders in question and repair them manually. Move settings from $SPLUNK_HOME/etc/system/local to custom apps. Ensure they are consulting the DS and are represented in a serverclass on the DS.

topic Mysterious Log Feed in Getting Data In

Mysterious Log Feed

Re: Mysterious Log Feed