All Apps and Add-ons

DMC returning duplicate results for all searches

sylim_splunk
Splunk Employee
Splunk Employee

When running searches on the Monitoring Console, we are getting duplicates...Pretty much every result is doubled.  

Env: 

DMC + 10 SHs in SHC + 230 Indexers in Cluster (RF=3, SF=2)

DMC with 7.3.3 + 16GB + 12 Cores

 

Below is the one we found the symptoms first time. 

index=_internal source=*license_usage.log* type=RolloverSummary   against past 7days/

When I run the same search on a production SH it gives exactly 1/2 in usage which we believe correct. Why is it happening to DMC?

 

Labels (2)
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

It turned out to be caused by wrong configurations  in DMC, especially for the Indexer Cluster.  It's supposed to be configured as a search head for the clustering.

Check the server.conf  and make sure to have the below;

[clustering]  

mode=searchhead

Why it happens:

- Instead of that, the DMC has added all the indexers under the stanza, 'distributedSearch' which means, indexers in cluster are  individual indexer for distributed searching. Since it has 2 for search factors all the searchable buckets including replicated searchable buckets returned the RolloverSummary data and gets double the results.

- Even if the DMC is configured as a SH under clustering stanza,  if all the idexers are listed under "distributedSearch" then  you will get the duplicate results too.

 

To fix this,

1. delete all the indexers listed in the stanza "[distributedSearch]" in distsearch.conf but leave CM - do not delete it.

2. add the DMC as a search head for the cluster

  ./splunk edit clustering-config -mode searchhead -master_uri https://<master>:8089 -site site0 -secret <cluster_secret>

3. restart the DMC.

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

It turned out to be caused by wrong configurations  in DMC, especially for the Indexer Cluster.  It's supposed to be configured as a search head for the clustering.

Check the server.conf  and make sure to have the below;

[clustering]  

mode=searchhead

Why it happens:

- Instead of that, the DMC has added all the indexers under the stanza, 'distributedSearch' which means, indexers in cluster are  individual indexer for distributed searching. Since it has 2 for search factors all the searchable buckets including replicated searchable buckets returned the RolloverSummary data and gets double the results.

- Even if the DMC is configured as a SH under clustering stanza,  if all the idexers are listed under "distributedSearch" then  you will get the duplicate results too.

 

To fix this,

1. delete all the indexers listed in the stanza "[distributedSearch]" in distsearch.conf but leave CM - do not delete it.

2. add the DMC as a search head for the cluster

  ./splunk edit clustering-config -mode searchhead -master_uri https://<master>:8089 -site site0 -secret <cluster_secret>

3. restart the DMC.

View solution in original post

thambisetty
Super Champion

This is very common mistake committed by admins.

————————————
If this helps, give a like below.
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.