Decomissioning indexers

[brent_weaver]

I am working to decommission some indexers from my cluster. I am using splunk offline --enforce-counts and letting them decommission on their own. I tried to do 8 at a time and it would not let me, but I could do one or two at a time and the others would just not decommission.

Any thoughts? Do I need to put the CM in maintainance mode?

[ivanreis]

It takes sometime until the indexer peer is completely stopped.

Here is how the splunk offline command works:

splunk offline: This is the fast version version of the splunk offline command. The peer goes down quickly, after a maximum of five minutes, even if searches or remedial activities are still in progress.
splunk offline --enforce-counts: This is the enforce-counts version of the command, which is designed to validate that the cluster has returned to the complete state. If you invoke the enforce-counts flag, the peer does not go down until all remedial activities are complete.

Further information -> https://docs.splunk.com/Documentation/Splunk/8.0.1/Indexer/Whathappenswhenaslavenodegoesdown

My suggestions is to remove the indexers from the cluster individually. Set the CM in maintenance mode and run "splunk offline" on the indexer you to remove.

Set CM in maintenance mode means that buckets hot buckets will not run on issues to roll up the bucket, so it is safe and recommended to setup this feature when you are doing any type of maintenance on the indexers.
More information about maintenance mode check this document -> https://docs.splunk.com/Documentation/Splunk/8.0.1/Indexer/Usemaintenancemode

[Akeydel]

If you are decommissioning indexers, they have to redistribute all the data on them to other peers in the cluster. If you try to take down several all at once, that process will likely break.

Definitely decom them individually.

Also, if you're using  ./splunk offline --enforce-counts DO NOT set maintenance mode

• The cluster cannot be in maintenance mode, because bucket fixup does not occur during maintenance mode.

https://docs.splunk.com/Documentation/Splunk/9.3.0/Indexer/Takeapeeroffline#The_enforce-counts_offli...

---