Getting Data In

What happens to the data if the indexer in an indexer cluster goes down?

sreejith2k2
Explorer

I have 12 Indexers (6 each/site) in a multi cluster environment. Data is replicated to the other site with RF =2 and SF =2, so even if one indexer (INDX01) goes down due to network issues, the indexer which is down currently holds data will be replicated to the other available 5 servers.

My question is, what happens to the data once that Indexer INDX01 is back (say after 2 days) to the cluster?

Will the indexer (INDX02 - 05) servers will start replicating the same data which is already there into the indexer (INDX01)? If so, will it have 3 copies, or it will delete the data?

Also, what is the case, if I enable maintenance mode once I come to know that the server is going to be offline for more than 24hrs, and what if I haven't enabled the maintenance mode?

0 Karma
1 Solution

ChrisG
Splunk Employee
Splunk Employee

There are also extensive docs on this subject, see What happens when a peer node goes down in the Managing Indexers and Clusters of Indexers manual.

See also:

View solution in original post

ChrisG
Splunk Employee
Splunk Employee

There are also extensive docs on this subject, see What happens when a peer node goes down in the Managing Indexers and Clusters of Indexers manual.

See also:

sreejith2k2
Explorer

Thanks Chris

0 Karma

botkindl
Explorer

When INDX01 comes back on line, it sends a list of all of its buckets to the cluster master. The cluster may end up with excess buckets, which you can remove from the master's UI.

Bear in mind that the indexers won't replicate on their own, unless the master is down. The master tells the indexers which buckets to replicate and where to send them. So, the other indexers won't send data to INDX01 that it already has. If INDX01 needs to have buckets to satisfy SF or RF, the master will tell the other indexers to replicate as needed.

If you do not enable maintenance mode while INDX01 is down, the master will tell your remaining indexers to replicate all of the buckets that INDX01 had when it went down -- in order to satisfy SF and RF. That can cause issues if you're short on disk space. If you do enable maintenance mode, the SF and RF are not enforced.

0 Karma

sreejith2k2
Explorer

Thanks botkindl

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...