DMC returning duplicate results for all searches

[sylim_splunk]

When running searches on the Monitoring Console, we are getting duplicates...Pretty much every result is doubled.  

Env: 

DMC + 10 SHs in SHC + 230 Indexers in Cluster (RF=3, SF=2)

DMC with 7.3.3 + 16GB + 12 Cores

Below is the one we found the symptoms first time. 

index=_internal source=*license_usage.log* type=RolloverSummary   against past 7days/

When I run the same search on a production SH it gives exactly 1/2 in usage which we believe correct. Why is it happening to DMC?

[sylim_splunk · Accepted solution]

It turned out to be caused by wrong configurations  in DMC, especially for the Indexer Cluster.  It's supposed to be configured as a search head for the clustering.

Check the server.conf  and make sure to have the below;

[clustering]  

mode=searchhead

Why it happens:

- Instead of that, the DMC has added all the indexers under the stanza, 'distributedSearch' which means, indexers in cluster are  individual indexer for distributed searching. Since it has 2 for search factors all the searchable buckets including replicated searchable buckets returned the RolloverSummary data and gets double the results.

- Even if the DMC is configured as a SH under clustering stanza,  if all the idexers are listed under "distributedSearch" then  you will get the duplicate results too.

To fix this,

1. delete all the indexers listed in the stanza "[distributedSearch]" in distsearch.conf but leave CM - do not delete it.

2. add the DMC as a search head for the cluster

  ./splunk edit clustering-config -mode searchhead -master_uri https://<master>:8089 -site site0 -secret <cluster_secret>

3. restart the DMC.

[thambisetty]

This is very common mistake committed by admins.