Splunk Enterprise

Search affinity and data replication in multisite clustering results in inconsistent search results

Lucas_K
Motivator

Take the following simple architecture.

2 indexer sites. 1 peer per site. Indexer A and Indexer B
2 search head sites. 1 sh per site. 1 sh has site affinity search head "fred" the other does not, search head "barney".
1 forwarder which auto load balances between sites based on a 1 min timer.

If we search for our events from fred. We will see events from indexer A only.
If we search for our events from barney. We will see events from indexer A and B depending on where the data landed.

The problem is that if we had two browser windows open and searches on both search heads are executed within 1 min of each other the number of events returned will differ. If I rerun the search from fred 2-3 mins later then all events will be visible.

This search event availability latency is a problem.

My thoughts on what is occurring is that the hot event streaming from site B to site A index is occurring slowly (much greater than a few seconds).

The quick solution is to disable search affinity so that events are pulled from their landing indexer.

My question however, is how can I re mediate this (without disabling affinity) AND how can I detect how long this search availability latency is?

0 Karma

deepashri_123
Motivator

Hey Lucas K,

Since your data is load balanced on both the site (indexers), both indexers will have primary copies of data.
So when you add search affinity at any point it will search the primary copies on the indexers and hence the results might have latency.
Search affinity can be added if you have all the primary copies on 1 site and secondary copies on other side, which will search for results on site1 only.
In your scenario, its better if you disable search affinity
Reference:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/Multisitesearchaffinity
Let me know if this helps!!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...