Why am I getting duplicate results after adding indexer cluster to distributed search?

[aferone]

I am testing our new indexer cluster using our existing search head. I added the indexer cluster servers to "dist_search" and created an indexer group so I can search just the cluster. However, all of the logs I am searching are duplicated.

I am not sure if this is because of the dist_search settings I have, a mis-configuration of the index cluster, or something else.

I did have load balancing set up on the heavy forwarders, sending to all 3 indexers. I removed that config and am only sending to 1 indexer, but the events are still duplicated.

Please help! Thanks!

[renjith_nair · Accepted solution]

From http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Configurethesearchhead

How the Distributed Search page works with indexer clusters
Do not use the Distributed Search page on Splunk Web to configure a search head in an indexer cluster or to add peers to the cluster. You can, however, use that page to view the list of search peers.
You are now replicating the data across your indexer cluster and then search on each node which creates the duplicates.

Instead of configuring each node of your indexer cluster as a distributed search node, you should configure the search head to connect to master which gives the search head the respective indexer to search for your data.
See Enablethesearchhead for enabling search head in an indexer cluster environment.

[aferone]

FIxed! This was it! Thank you SO MUCH!

[aferone]

This looks promising. I am going to set up our new search head cluster environment first, then try this. Thanks for the answer, and I will accept if I can get it working!

[gcusello]

Verify outputs.conf in your heavy forwarders if you correctly configured autoloadbalancing (in https://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Outputsconf see example autoloadbalancing).

About the test to send events to only one indexer: duplicated events are the new ones or only the old ones? because you cannot modify the indexed events, only the new ones.

Bye.
Giuseppe

[aferone]

I cleared out the index by setting frozentimeperiodinsecs to a low number, rolling all of the logs to frozen, then setting the time period back to the normal retention, to start the index clean. Then, I set up the Heavy Forwarder to only send to one indexer. Si I believe I removed the Heavy Forwarder from being the issue.