topic Re: Replacing search peer in an indexer cluster - Best practices/concerns in Deployment Architecture

Replacing search peer in an indexer cluster - Best practices/concerns

datlaphani — Tue, 29 Sep 2020 15:48:43 GMT

Hi Splunk experts,

We have a 2 site index cluster with 2 indexers per site. The plan is to replace existing disks on the indexers to allocate more space on one indexer at a time. Our current SF and RF setting are below:
multisite=true
available_sites=site1,site2
site_replication_factor = origin:2,total:3
site_search_factor = origin:1,total:2

Current disk utilization:
Site1: indexer1 - 90%,indexer2 - 62%
Site2: indexer1 - 83%,indexer2 - 42%

Question1:
what is the best way to do this activity?
Run the splunk offline --enforce-counts on one of the indexers, wait for the data to redistribute, complete the drive upgrades, reinstall splunk and re-add the peer to the cluster. Repeat the same on all the indexers.

Question2:
During this activity, as the replication factor will not be met, does it affect anything?

Question3:
If I bring the indexer1 - 90% offline, will the space on indexer2 - 62% be sufficient to generate the searchable copies?

Re: Replacing search peer in an indexer cluster - Best practices/concerns

koshyk — Wed, 20 Sep 2017 13:21:33 GMT

Luckily your environment is small in count for indexers. Best way to do is

Question1: what is the best way to do this activity?
- Put Splunk into maintenance mode. This means indexers won't replicate. Then stop splunk on one indexer per site. Add drives/upgrade etc. and start it back. After everything is done, disable maintenance mode and it will start replication

Question2: During this activity, as the replication factor will not be met, does it affect anything?
It depends on the criticality of your environment. If you Search Head have cross site search facility then the end-users won't see any impact. For the upgrade duration, the only risk is your redudancy is impacted.

Question3: If I bring the indexer1 - 90% offline, will the space on indexer2 - 62% be sufficient to generate the searchable copies?
Best thing to do in your case is upgrade site1-indexer1 first , so when you bring it back it have enough storage. Then site2-indexer1 and so on..

Re: Replacing search peer in an indexer cluster - Best practices/concerns

datlaphani — Wed, 20 Sep 2017 19:27:11 GMT

Hi Koshyk, Thanks for the answers.
- Put Splunk into maintenance mode. This means indexers won't replicate. Then stop splunk on one indexer per site. Add drives/upgrade etc. and start it back. After everything is done, disable maintenance mode and it will start replication

As part of the dive upgrades, we will need to re-image the system, as the dives are going to be completely replaced. We are trying to figure out the best way to do this activity without affecting users. so maintenance mode may not work.