[Multiple Clusters] search not returning from new cluster

[sylim_splunk]

A standalone SH had 2 multiple indexer clusters configured as per the docs below, which has been working well a while until I added 3rd cluster.

https://docs.splunk.com/Documentation/Splunk/7.2.4/Indexer/Configuremulti-clustersearch

The searches appear working only across the existing 2 indexer clusters but the indexers from the 3rd cluster do not return any data.

[clustering]
mode = searchhead
master_uri = clustermaster:one, clustermaster:two, clustermaster:three

[clustermaster:three]
master_uri=https://SplunkMaster03.com:8089
pass4SymmKey=correctSecret

Please help.

[sylim_splunk · Accepted solution]

Search.log in Search job inspector shows that searches were only dispatched to the 2 existing cluster indexers but not to the new indexers.

The search GUI shows the field name, splunk_server_group field shows that dmc_xxx groups. This means the deployment monitoring console, DMC is enabled on the search head. With DMC enabled on SH, only search peers that are members of the indexer group are searched by default. Because all searches that run on the monitoring console instance follow this behavior, non-monitoring console searches might have incomplete results.

"Do Not Use Production Search Head for DMC" which is also documented in the link below - Resolution: Switching to "standalone" next to "distributed" fixed the issue.

https://docs.splunk.com/Documentation/Splunk/7.2.4/DMC/WheretohostDMC