Replacing search peer in an indexer cluster - Best practices/concerns

[datlaphani]

Hi Splunk experts,

We have a 2 site index cluster with 2 indexers per site. The plan is to replace existing disks on the indexers to allocate more space on one indexer at a time. Our current SF and RF setting are below:
multisite=true
available_sites=site1,site2
site_replication_factor = origin:2,total:3
site_search_factor = origin:1,total:2

Current disk utilization:
Site1: indexer1 - 90%,indexer2 - 62%
Site2: indexer1 - 83%,indexer2 - 42%

Question1:
what is the best way to do this activity?
Run the splunk offline --enforce-counts on one of the indexers, wait for the data to redistribute, complete the drive upgrades, reinstall splunk and re-add the peer to the cluster. Repeat the same on all the indexers.

Question2:
During this activity, as the replication factor will not be met, does it affect anything?

Question3:
If I bring the indexer1 - 90% offline, will the space on indexer2 - 62% be sufficient to generate the searchable copies?

[koshyk]

Luckily your environment is small in count for indexers. Best way to do is

Question1: what is the best way to do this activity?
- Put Splunk into maintenance mode. This means indexers won't replicate. Then stop splunk on one indexer per site. Add drives/upgrade etc. and start it back. After everything is done, disable maintenance mode and it will start replication

Question2: During this activity, as the replication factor will not be met, does it affect anything?
It depends on the criticality of your environment. If you Search Head have cross site search facility then the end-users won't see any impact. For the upgrade duration, the only risk is your redudancy is impacted.

Question3: If I bring the indexer1 - 90% offline, will the space on indexer2 - 62% be sufficient to generate the searchable copies?
Best thing to do in your case is upgrade site1-indexer1 first , so when you bring it back it have enough storage. Then site2-indexer1 and so on..

[datlaphani]

Hi Koshyk, Thanks for the answers.
- Put Splunk into maintenance mode. This means indexers won't replicate. Then stop splunk on one indexer per site. Add drives/upgrade etc. and start it back. After everything is done, disable maintenance mode and it will start replication

As part of the dive upgrades, we will need to re-image the system, as the dives are going to be completely replaced. We are trying to figure out the best way to do this activity without affecting users. so maintenance mode may not work.