we have a s.o.s app to check the health of the indexer pool, i recently find out that one indexer is fully loaded on parsing, aggregation and queue for indexing but the other one is almost empty. it extremely not balance. Does anyone know how do I begin to troubleshoot that?
I am assuming you are using Splunk forwarders with auto-load-balance to get data into your indexers.
This is a really good answer on this subject from @lguinn:
Describe your datasources, please If you have extremely busy ones, without any sort of "pause" i.e. syslog with very consistent stream of incoming data, the UF may never "switch" to another indexer using the auto-load-balancing feature. This is because the UF does not recognize event boundaries, so it waits for an EOF before it switches to another indexer. See here: