My searches are failing on Search Head with error " Search filters specified using splunk_server/splunk_server_group do not match any search peer"

[sat94541]

I am using Splunk Version 6.2.2 with search head and few indexer. My searches are failing with following error

Search filters specified using splunk_server/splunk_server_group do not match any search peer.

[rbal_splunk]

The messages you are seeing is due to new App called DMC ( Refer http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/ConfiguretheMonitoringConsole) that is installed with Splunk Version 6.2.

If you run the btool for distsearch on the Search head

./splunk cmd btool distsearch list --debug

You will see multiple groups

..../etc/system/local/distsearch.conf [distributedSearch]
……..
..../etc/system/local/distsearch.conf servers = index500010:8089,index500011:8089,index500012:8089,index500013:8089
..../etc/system/default/distsearch.conf shareBundles = true
..../etc/system/default/distsearch.conf statusTimeout = 10
..../etc/system/default/distsearch.conf useSHPBundleReplication = true
..../etc/system/local/distsearch.conf [distributedSearch:dmc_group_cluster_master]
..../etc/system/local/distsearch.conf [distributedSearch:dmc_group_deployment_server]
..../etc/system/local/distsearch.conf [distributedSearch:dmc_group_indexer]
..../etc/system/local/distsearch.conf default = true
..../etc/system/local/distsearch.conf [distributedSearch:dmc_group_kv_store]

Notice that distsearch.conf has many groups and the group [distributedSearch:dmc_group_indexer] is marked as default group using "default = true"

Here group [distributedSearch] has list of indexer to search , but default group [distributedSearch:dmc_group_indexer] has no reference to any server(indexers).

To resolve this issue make the following change to the distsearch.conf

[distributedSearch]
default = true
[distributedSearch:dmc_group_indexer]
default = false

Restart splunk service after making this chnage.