All Apps and Add-ons

DMC returning duplicate results for all searches

sylim_splunk
Splunk Employee
Splunk Employee

When running searches on the Monitoring Console, we are getting duplicates...Pretty much every result is doubled.  

Env: 

DMC + 10 SHs in SHC + 230 Indexers in Cluster (RF=3, SF=2)

DMC with 7.3.3 + 16GB + 12 Cores

 

Below is the one we found the symptoms first time. 

index=_internal source=*license_usage.log* type=RolloverSummary   against past 7days/

When I run the same search on a production SH it gives exactly 1/2 in usage which we believe correct. Why is it happening to DMC?

 

Labels (1)
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

It turned out to be caused by wrong configurations  in DMC, especially for the Indexer Cluster.  It's supposed to be configured as a search head for the clustering.

Check the server.conf  and make sure to have the below;

[clustering]  

mode=searchhead

Why it happens:

- Instead of that, the DMC has added all the indexers under the stanza, 'distributedSearch' which means, indexers in cluster are  individual indexer for distributed searching. Since it has 2 for search factors all the searchable buckets including replicated searchable buckets returned the RolloverSummary data and gets double the results.

- Even if the DMC is configured as a SH under clustering stanza,  if all the idexers are listed under "distributedSearch" then  you will get the duplicate results too.

 

To fix this,

1. delete all the indexers listed in the stanza "[distributedSearch]" in distsearch.conf but leave CM - do not delete it.

2. add the DMC as a search head for the cluster

  ./splunk edit clustering-config -mode searchhead -master_uri https://<master>:8089 -site site0 -secret <cluster_secret>

3. restart the DMC.

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

It turned out to be caused by wrong configurations  in DMC, especially for the Indexer Cluster.  It's supposed to be configured as a search head for the clustering.

Check the server.conf  and make sure to have the below;

[clustering]  

mode=searchhead

Why it happens:

- Instead of that, the DMC has added all the indexers under the stanza, 'distributedSearch' which means, indexers in cluster are  individual indexer for distributed searching. Since it has 2 for search factors all the searchable buckets including replicated searchable buckets returned the RolloverSummary data and gets double the results.

- Even if the DMC is configured as a SH under clustering stanza,  if all the idexers are listed under "distributedSearch" then  you will get the duplicate results too.

 

To fix this,

1. delete all the indexers listed in the stanza "[distributedSearch]" in distsearch.conf but leave CM - do not delete it.

2. add the DMC as a search head for the cluster

  ./splunk edit clustering-config -mode searchhead -master_uri https://<master>:8089 -site site0 -secret <cluster_secret>

3. restart the DMC.

thambisetty
SplunkTrust
SplunkTrust

This is very common mistake committed by admins.

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...