I am using Splunk Version 6.2.2 with search head and few indexer. My searches are failing with following error
Search filters specified using splunk_server/splunk_server_group do not match any search peer.
The messages you are seeing is due to new App called DMC ( Refer http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/ConfiguretheMonitoringConsole) that is installed with Splunk Version 6.2.
If you run the btool for distsearch on the Search head
./splunk cmd btool distsearch list --debug
You will see multiple groups
..../etc/system/local/distsearch.conf [distributedSearch]
……..
..../etc/system/local/distsearch.conf servers = index500010:8089,index500011:8089,index500012:8089,index500013:8089
..../etc/system/default/distsearch.conf shareBundles = true
..../etc/system/default/distsearch.conf statusTimeout = 10
..../etc/system/default/distsearch.conf useSHPBundleReplication = true
..../etc/system/local/distsearch.conf [distributedSearch:dmc_group_cluster_master]
..../etc/system/local/distsearch.conf [distributedSearch:dmc_group_deployment_server]
..../etc/system/local/distsearch.conf [distributedSearch:dmc_group_indexer]
..../etc/system/local/distsearch.conf default = true
..../etc/system/local/distsearch.conf [distributedSearch:dmc_group_kv_store]
Notice that distsearch.conf has many groups and the group [distributedSearch:dmc_group_indexer] is marked as default group using "default = true"
Here group [distributedSearch] has list of indexer to search , but default group [distributedSearch:dmc_group_indexer] has no reference to any server(indexers).
To resolve this issue make the following change to the distsearch.conf
[distributedSearch]
default = true
[distributedSearch:dmc_group_indexer]
default = false
Restart splunk service after making this chnage.