I've come up with a query to see which indexers our forwarders are sending data to and the results are somewhat eye opening. I'd be interested in other sets of eyes on the logic of my query and to see if others are experiencing the same. The idea is to use the internal logs from the forwarders keying in on the phrase "connected to idx". The query is
index=_internal "connected to idx" | rex field=idx "(?<indexer>[^:]+)" | stats count by indexer | sort indexer
I'm sorting the indexers in numeric order (IP) because that is the way they are listed in the outputs.conf. What is interesting is the visual profile of the distribution is basically the same regardless of the time of day or time span of the query. In other words my first indexer receives about 2x the connections as the others whether that is in the last 5 minutes or 7 days. The same visual profile regardless of time/time range is generally true for the others. I've come up with a query to match the results of this data to the GB indexed per indexer in the same time period but it needs to be run in the deployment monitor app - wasn't going to rewrite all the macros. I also had to do some heavy massaging in order to match IP to indexer name and then to sort correctly. You might not have that issue /shrug. Run it in its base form and see (I've left out post append stuffs)
index=_internal "connected to idx" | rex field=idx "(?<indexer>[^:]+)" | eval "Splunk Server" = indexer | stats count by "Splunk Server" | append [search `all_indexers_summary`] | sort "Splunk Server"
Environment
20 indexers
1300 forwarders
Using auto load balancing built into outputs.conf
Forwarder versions range from 4.3.4 to 6.0.1
