Why are we getting different results from search heads in our Splunk 6.2.3 search head and multisite indexer clustering environment? Is this a replication issue?

[jmallorquin]

Hi,

We have an implementation of 3 search heads in a search head cluster and 4 indexers in a multisite indexer cluster all in Splunk 6.2.3.
The search heads have search affinity (sh1 and sh2 site1 and sh3 site2).
We have detected that doing the same search across the search head, we have different results. It looks like there is a problem with the replication across the sites.

Checking the DMC all the index looks ok.

Why this problem is not shown in the DMC?
Is there a way to rebuild the index to the site that has the replication problem?

Thanks,

[jmallorquin · Accepted solution]

In the process of troubleshooting we install S.o.S in the master and when we apply the restart to finish the instalation, the problem fixed.

Thanks,

[javiergn]

It might be worth taking a look at the Indexer Clustering dashboard and verifying that the bucket replication status is correct.
There's also a very detailed page on troubleshooting replication issues here.

Hope that helps

[tom_frotscher]

If you have active inputs, it might be correct to get different results if you for example search over the "last 4 hours". You need a static timerange, that can't "change" while you do your search. For example do your searches for yesterday. Did you consider this in your current analysis?