All Apps and Add-ons

DMC returning duplicate results for all searches

sylim_splunk
Splunk Employee
Splunk Employee

When running searches on the Monitoring Console, we are getting duplicates...Pretty much every result is doubled.  

Env: 

DMC + 10 SHs in SHC + 230 Indexers in Cluster (RF=3, SF=2)

DMC with 7.3.3 + 16GB + 12 Cores

 

Below is the one we found the symptoms first time. 

index=_internal source=*license_usage.log* type=RolloverSummary   against past 7days/

When I run the same search on a production SH it gives exactly 1/2 in usage which we believe correct. Why is it happening to DMC?

 

Labels (2)
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

It turned out to be caused by wrong configurations  in DMC, especially for the Indexer Cluster.  It's supposed to be configured as a search head for the clustering.

Check the server.conf  and make sure to have the below;

[clustering]  

mode=searchhead

Why it happens:

- Instead of that, the DMC has added all the indexers under the stanza, 'distributedSearch' which means, indexers in cluster are  individual indexer for distributed searching. Since it has 2 for search factors all the searchable buckets including replicated searchable buckets returned the RolloverSummary data and gets double the results.

- Even if the DMC is configured as a SH under clustering stanza,  if all the idexers are listed under "distributedSearch" then  you will get the duplicate results too.

 

To fix this,

1. delete all the indexers listed in the stanza "[distributedSearch]" in distsearch.conf but leave CM - do not delete it.

2. add the DMC as a search head for the cluster

  ./splunk edit clustering-config -mode searchhead -master_uri https://<master>:8089 -site site0 -secret <cluster_secret>

3. restart the DMC.

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

It turned out to be caused by wrong configurations  in DMC, especially for the Indexer Cluster.  It's supposed to be configured as a search head for the clustering.

Check the server.conf  and make sure to have the below;

[clustering]  

mode=searchhead

Why it happens:

- Instead of that, the DMC has added all the indexers under the stanza, 'distributedSearch' which means, indexers in cluster are  individual indexer for distributed searching. Since it has 2 for search factors all the searchable buckets including replicated searchable buckets returned the RolloverSummary data and gets double the results.

- Even if the DMC is configured as a SH under clustering stanza,  if all the idexers are listed under "distributedSearch" then  you will get the duplicate results too.

 

To fix this,

1. delete all the indexers listed in the stanza "[distributedSearch]" in distsearch.conf but leave CM - do not delete it.

2. add the DMC as a search head for the cluster

  ./splunk edit clustering-config -mode searchhead -master_uri https://<master>:8089 -site site0 -secret <cluster_secret>

3. restart the DMC.

thambisetty
SplunkTrust
SplunkTrust

This is very common mistake committed by admins.

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...