Getting Data In

Decomissioning indexers

brent_weaver
Builder

I am working to decommission some indexers from my cluster. I am using splunk offline --enforce-counts and letting them decommission on their own. I tried to do 8 at a time and it would not let me, but I could do one or two at a time and the others would just not decommission.

Any thoughts? Do I need to put the CM in maintainance mode?

0 Karma

ivanreis
Builder

It takes sometime until the indexer peer is completely stopped.

Here is how the splunk offline command works:

splunk offline: This is the fast version version of the splunk offline command. The peer goes down quickly, after a maximum of five minutes, even if searches or remedial activities are still in progress.
splunk offline --enforce-counts: This is the enforce-counts version of the command, which is designed to validate that the cluster has returned to the complete state. If you invoke the enforce-counts flag, the peer does not go down until all remedial activities are complete.

Further information -> https://docs.splunk.com/Documentation/Splunk/8.0.1/Indexer/Whathappenswhenaslavenodegoesdown

My suggestions is to remove the indexers from the cluster individually. Set the CM in maintenance mode and run "splunk offline" on the indexer you to remove.

Set CM in maintenance mode means that buckets hot buckets will not run on issues to roll up the bucket, so it is safe and recommended to setup this feature when you are doing any type of maintenance on the indexers.
More information about maintenance mode check this document -> https://docs.splunk.com/Documentation/Splunk/8.0.1/Indexer/Usemaintenancemode

Akeydel
Explorer

If you are decommissioning indexers, they have to redistribute all the data on them to other peers in the cluster. If you try to take down several all at once, that process will likely break.

Definitely decom them individually.

Also, if you're using  ./splunk offline --enforce-counts DO NOT set maintenance mode

  • The cluster cannot be in maintenance mode, because bucket fixup does not occur during maintenance mode.
https://docs.splunk.com/Documentation/Splunk/9.3.0/Indexer/Takeapeeroffline#The_enforce-counts_offli...
0 Karma
Get Updates on the Splunk Community!

New Case Study Shows the Value of Partnering with Splunk Academic Alliance

The University of Nevada, Las Vegas (UNLV) is another premier research institution helping to shape the next ...

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...