When running searches on the Monitoring Console, we are getting duplicates...Pretty much every result is doubled.
Env:
DMC + 10 SHs in SHC + 230 Indexers in Cluster (RF=3, SF=2)
DMC with 7.3.3 + 16GB + 12 Cores
Below is the one we found the symptoms first time.
index=_internal source=*license_usage.log* type=RolloverSummary against past 7days/
When I run the same search on a production SH it gives exactly 1/2 in usage which we believe correct. Why is it happening to DMC?
It turned out to be caused by wrong configurations in DMC, especially for the Indexer Cluster. It's supposed to be configured as a search head for the clustering.
Check the server.conf and make sure to have the below;
[clustering]
mode=searchhead
Why it happens:
- Instead of that, the DMC has added all the indexers under the stanza, 'distributedSearch' which means, indexers in cluster are individual indexer for distributed searching. Since it has 2 for search factors all the searchable buckets including replicated searchable buckets returned the RolloverSummary data and gets double the results.
- Even if the DMC is configured as a SH under clustering stanza, if all the idexers are listed under "distributedSearch" then you will get the duplicate results too.
To fix this,
1. delete all the indexers listed in the stanza "[distributedSearch]" in distsearch.conf but leave CM - do not delete it.
2. add the DMC as a search head for the cluster
./splunk edit clustering-config -mode searchhead -master_uri https://<master>:8089 -site site0 -secret <cluster_secret>
3. restart the DMC.
It turned out to be caused by wrong configurations in DMC, especially for the Indexer Cluster. It's supposed to be configured as a search head for the clustering.
Check the server.conf and make sure to have the below;
[clustering]
mode=searchhead
Why it happens:
- Instead of that, the DMC has added all the indexers under the stanza, 'distributedSearch' which means, indexers in cluster are individual indexer for distributed searching. Since it has 2 for search factors all the searchable buckets including replicated searchable buckets returned the RolloverSummary data and gets double the results.
- Even if the DMC is configured as a SH under clustering stanza, if all the idexers are listed under "distributedSearch" then you will get the duplicate results too.
To fix this,
1. delete all the indexers listed in the stanza "[distributedSearch]" in distsearch.conf but leave CM - do not delete it.
2. add the DMC as a search head for the cluster
./splunk edit clustering-config -mode searchhead -master_uri https://<master>:8089 -site site0 -secret <cluster_secret>
3. restart the DMC.
This is very common mistake committed by admins.