What happens to the data if the indexer in an indexer cluster goes down?

[sreejith2k2]

I have 12 Indexers (6 each/site) in a multi cluster environment. Data is replicated to the other site with RF =2 and SF =2, so even if one indexer (INDX01) goes down due to network issues, the indexer which is down currently holds data will be replicated to the other available 5 servers.

My question is, what happens to the data once that Indexer INDX01 is back (say after 2 days) to the cluster?

Will the indexer (INDX02 - 05) servers will start replicating the same data which is already there into the indexer (INDX01)? If so, will it have 3 copies, or it will delete the data?

Also, what is the case, if I enable maintenance mode once I come to know that the server is going to be offline for more than 24hrs, and what if I haven't enabled the maintenance mode?

[ChrisG · Accepted solution]

There are also extensive docs on this subject, see What happens when a peer node goes down in the Managing Indexers and Clusters of Indexers manual.

See also:

• What happens when a peer node comes back up
• What happens when the master node goes down

[sreejith2k2]

Thanks Chris

[botkindl]

When INDX01 comes back on line, it sends a list of all of its buckets to the cluster master. The cluster may end up with excess buckets, which you can remove from the master's UI.

Bear in mind that the indexers won't replicate on their own, unless the master is down. The master tells the indexers which buckets to replicate and where to send them. So, the other indexers won't send data to INDX01 that it already has. If INDX01 needs to have buckets to satisfy SF or RF, the master will tell the other indexers to replicate as needed.

If you do not enable maintenance mode while INDX01 is down, the master will tell your remaining indexers to replicate all of the buckets that INDX01 had when it went down -- in order to satisfy SF and RF. That can cause issues if you're short on disk space. If you do enable maintenance mode, the SF and RF are not enforced.

[sreejith2k2]

Thanks botkindl