Can we decommission multiple indexers at a time from one site in multi-site indexer clustering?

[arjunagarwal]

Hi All,

Currently, i'm decommissioning a few indexers in a multi-site indexer cluster with 2 sites having RF=3 & SF=2 . Right now, there are a total of 100 indexers with 50 indexers each per site.

I need to decommission 60 indexers (30 from each site). The plan is to put the 60 indexers in detention mode and make one peer offline by using enforce counts at a time.

In order to reduce the duration of the entire decommissioning, can we put multiple peers offline at a time from a site without impacting the search consistency?

If not, then i would like to know your thoughts on the impact of decommissioning more than 1 indexer per site at a time.

Thanks in advance,

[inventsekar]

may we know your daily license(wondering about the 50 indexers per site!)

• I need to decommission 60 indexers 30 from each site* i hope better to decommission, lets say 3 or 6 from each site in parallel. if you decommission only on one site, it will surely impact the SF and RF (or the search performance will get impacted)

The plan is to put the 60 indexers in detention mode and make one peer offline by using enforce counts at a time

Once a server is put into detention, it is essentially removed from the cluster in terms of bucket replication and rebalancing. So if your search / replication factor is no longer met with these servers in detention, then yes your SF wont be made and you will get these errors.

Manually rebalancing should work assuming you have the requisite number of peers in your site SF/RF. See this doc : http://docs.splunk.com/Documentation/Splunk/6.3.3/Indexer/Rebalancethecluster

From - https://answers.splunk.com/answers/369782/does-manually-enforcing-detention-mode-in-a-multis-1.html

[arjunagarwal]

Daily Licensing is ~4.5 TB/day & around 170 correlation searches with ES.
We are migrating from old 60 indexers m4.10xlarge to new 40 i3.16xlarge. I've already added 40 new indexers(i3.16xlarge) with 20 on each site & data rebalancing on the cluster is also complete with SF/RF met.

Now i need to decommission old 60 m4.10xlarge from the indexer cluster. I would like to know your thoughts for removing these indexers from the cluster without any impact on data & search performance such that we can remove multiple indexers at a time.

Removing indexers in parallel from both the site may cause loosing both the primary copies as SF=2&RF=3 is configured & will generate inconsistent search result.

[inventsekar]

I need to decommission 60 indexers (30 from each site)
May we know if you have already done this?!?!
As you said, no need to remove in parallel. Put 5 indexers from one site into detention. Wait for the SF/RF to be met. Then do the same to another 5 indexers at the opposite site. Wait for SF/RF to be met. Continue the same. On a weekend this can be completed i think.

If this resolved your query, pls accept this as the answer.