You could easily run a CLI or API search that checks on event counts from each Splunk Server. Obviously, there will be a penalty of running the search. Similarly, you could check on known sources across all servers and report on those event counts by "splunk_server". For example, to check all servers to see if they are indexing I could do this search:
index=_internal source=*metrics.log earliest=-2m | stats count by splunk_server
Regarding auto load balancing, it works as follows... Assume I have 5 indexers, call them i1, i2, i3, i4, and i5. If my forwarder is running autoLB, it will do the following:
Randomly select and indexer from the pool i1 - i5. In this case, let's assume we select i1.
Send data to i1 for the autoLB period.
Randomly select an indexer from the pool i2 - i5 (the indexer we previously sent data to is removed from the pool). In this case, let's assume we select i2.
If a connection is established, send data to i2 for the autoLB period.
If we cannot establish a connection, select another indexer from the pool i3-i5 (i2 is removed because we can't send data there, but i1 is also removed since we last sent data there). Let's assume we select i5...
Send data to i5 for the autoLB period
Randomly select an indexer from the pool i1-i4. Wash rinse repeat
Notice that splunk removes the previously used indexer from the pool, and if the subsequent indexer fails, it is also removed from the pool until success.
... View more