Search affinity and data replication in multisite clustering results in inconsistent search results

[Lucas_K]

Take the following simple architecture.

2 indexer sites. 1 peer per site. Indexer A and Indexer B
2 search head sites. 1 sh per site. 1 sh has site affinity search head "fred" the other does not, search head "barney".
1 forwarder which auto load balances between sites based on a 1 min timer.

If we search for our events from fred. We will see events from indexer A only.
If we search for our events from barney. We will see events from indexer A and B depending on where the data landed.

The problem is that if we had two browser windows open and searches on both search heads are executed within 1 min of each other the number of events returned will differ. If I rerun the search from fred 2-3 mins later then all events will be visible.

This search event availability latency is a problem.

My thoughts on what is occurring is that the hot event streaming from site B to site A index is occurring slowly (much greater than a few seconds).

The quick solution is to disable search affinity so that events are pulled from their landing indexer.

My question however, is how can I re mediate this (without disabling affinity) AND how can I detect how long this search availability latency is?

[deepashri_123]

Hey Lucas K,

Since your data is load balanced on both the site (indexers), both indexers will have primary copies of data.
So when you add search affinity at any point it will search the primary copies on the indexers and hence the results might have latency.
Search affinity can be added if you have all the primary copies on 1 site and secondary copies on other side, which will search for results on site1 only.
In your scenario, its better if you disable search affinity
Reference:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/Multisitesearchaffinity
Let me know if this helps!!