search comparison to validate all forwarders point to indexers in the idx cluster

[fisuser1]

GM, through the years we have added several indexers to our cluster. we are no looking to retire a few generation 1 indexers. does anyone have a search that can do a comparison of what forwarders are/are not reporting to certain indexers in the cluster by index name? perhaps a table to display all forwarder client servers not utilizing all indexers? my fear is we retire a set of gen 1 indexers and all forwarders are not reporting into ALL indexers so users will no longer see their data.

[adonio]

try this search and filter down the road as you see fit:
| tstats max(_time) as last_event where index=* by host index splunk_server
the splunk_server field is your indexers

[FrankVl]

But that doesn't really tell you anything about which forwarders are sending to which indexers, right? in some cases the host field will hold the forwarder name, but plenty of data feed architectures where the host field contains the original host and not the forwarder.

[adonio]

yup, but its a start ... you can approach it in many ways
you can do after the search ... stats values(splunk_server) as indexers by host or whatever
if you insist on that, you can use index=_* instead of index=* in the above search.
or you can use the field hostname or origianl_hostname whatever it might be

[adonio]

imho it will be easier to check your outputs.conf and modify the it according to your needs
also, have i mind that if your forwarders are setup correctly, they will keep on sending data to the new indexers

[FrankVl]

That is certainly also a good angle of attack, but that does depend a bit on how well managed the outputs.conf is. If there are all kinds of forwarders with different local configs or so, that becomes a bit more difficult. Getting a baseline from the actual logs where they are sending could then help identify forwarders that need attention.

[adonio]

good point, @FrankVl will put an answer with a search covering the desired results

[FrankVl]

The "by index" part might be a bit tricky (I also don't fully understand how that is relevant to be honest).

To see which forwarders are sending to which indexers, try something like this:

index=_internal sourcetype=splunkd group=tcpin_connections | stats count by host,hostname

Ideally add filters to limit the host field to only match your indexers and the hostname field to only match your forwarders.