Why can only the Search Head Captain search the Indexer Cluster?

[iacobeaj]

I have a search head cluster that is trying to search across an indexer cluster, but for some reason, only the Currently elected captain can search the indexers and get data back. All of the other members come back with the errors

"Gave up waiting for the captain to establish a common bundle version across all search peers; using most recent bundles on all peers instead." 

and

"Search filters specified using splunk_server/splunk_server_group do not match any search peer." 

When I do the search on the captain, I get results as I would expect.

[rbal_splunk]

inconsistent search peers across search head cluster members will cause search dispatch issues and the error message you see.

Check distsearch.conf on each of the Search Head Cluster Member something like

find */etc/{apps,system} -name distsearch.conf | grep local | xargs grep --color 'servers ='

Or else you can run btool for distsearch on each search head using command

$SPLUNK_HOME/bin/splunk cmd btool distsearch list --debug

and look for Server

number of Splunk Server should match on each Search Head Cluster Member.