Deployment Architecture

Indexer Replication Failover Testing

rturk
Builder

Hello Splunkers!

I have two clustered indexers (v5.0.4) replicating buckets between them. I have been testing the failover mechanism to tick a box saying that the data is indeed searchable if an indexer fails. The testing methodology is as follows:

  1. Run a search (e.g. index=main) for a window of time in the past.
  2. Confirm the number of results returned from each splunk_server, and the total number of events returned
  3. offline one of the indexers (IDX-A)
  4. Re-run the test for the same period of time to confirm identical results are returned.
  5. ???
  6. PROFIT!!

All pretty simple right? (apart from 5 & 6 anyway).

The problem I have is with step 4 - I do not get the results that were previously returned from the now offline indexer IDX-A. I have waited for ~10 minutes with no joy. I have a rep factor & search factor of 2, and the cluster master reports that everything is hunky-dory. But as soon as I restart the splunkd process on IDX-A, I get the correct/expected number of results from IDX-B. So yes, replication works... but I'd have expected the resumption of service on IDX-A not to be a trigger/catalyst for IDX-B actually returning events previously held by IDX-A.

Is there a setting/timer I'm missing here? Happy to be pointed in the right direction!

Thanks in advance 🙂

RT

0 Karma

svasan_splunk
Splunk Employee
Splunk Employee

R.Turk,

How much data did you index? (The replication is not per-event but per some amount of data. I'm wondering if the replication never happened before the shutdown.)

Also are you sending data via forwarder or local file via monitors? If forwarders, is your forwarder auto-lbing across all the peers? If the peer goes down without replicating the data, then the forwarder should just send the data to some other peer.

But if you are indexing local files via monitors that won't happen. You should be using forwarders that have acks turned on and set to auto-lb across all those peers.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...