Splunk 6.6.6 fixes this issue (edit: with no indication in release notes!). I can run clean installs with minimal, identical configs on 6.5.3, 6.6.3 and 6.6.6 and watch load balancing fail on larger clusters with the 6.6.3 version and work fine with the others. Entirely predictable. 6.6.3 eventually lands on just using 3 indexers over and over. No LB to the others.
This happens with SUF and full install.
If you're using 6.6.0-3 in an environment with lots of indexer targets, i'd strongly urge you to upgrade. ASAP.
On the MC, I run this search to find the suspected baddies:
`dmc_get_forwarder_tcpin` | stats values(fwdType) as fwdType, values(sourceIp) as sourceIp, latest(version) as version, values(os) as os, values(arch) as arch, p90(tcp_KBps) as avg_tcp_kbps, dc(splunk_server) as Indexers by hostname | where Indexers<6
Like clockwork, the only version with 3 indexers listed is 6.6.1-6.6.3 (i never deployed 6.6.0)
... View more