There's a few things that could be occurring here, so bare with me.
1) Are all the indexers same spec? RAM, CPU, and most importantly IO.. ( nvme / ssd / sas )
2) Are the search workloads pinning on any of these indexers? Is data balanced on across the indexers?
3) How many intermediates are you funneling the traffic through?
4) What's your autolb setting for the intermediates?
5) How many pipelines do you have on your intermediates?
Going through the above and thinking about Splunk specific configs (assuming equal hardware and sufficient resources..) you may have some bottlenecks with your intermediate tier funneling traffic.
Default autolb frequency is 30 seconds, have you adjusted this? Lowering this to 10s or 5s in large volume environments will help spread data more evenly across the indexing fleet. Additionally, how are the pipelines in your intermediate tier? Rule of thumb is you need 2 X # of indexing pipelines.
So in your case, you should have at least 60 pipelines in your intermediates to get the best event spread across your fleet.
Another point to check is if the internal logs are showing any timeouts, or connection refused, to these indexers in questions..
There are a few starting points.. Let us know how it goes.
