What are the next steps to prep servers for retirement?

[discenzadoe]

We have a distributed search environment, with 2 very old indexers (the original servers) and 3 new indexers in a cluster. 

The old indexers have been removed from the destination lists in outputs.conf nearly everywhere, and most of the data is between 5 and 6 months old, except for internal indexes.

I can't find what my next steps are to prep these servers for retirement, such as force-freezing the buckets they still hold, etc. 

Suggestions?

Thanks.

[richgalloway]

The first step is to remove the old indexers from outputs.conf *everywhere*, not just nearly.

The next step is to run the command splunk offline --enforce-counts on one indexer.  This will tell the cluster to make sure the buckets on the old indexer exist elsewhere in the cluster.  Then the indexer will stop itself.

The last step is to repeat the previous step on the remaining indexer.

See https://docs.splunk.com/Documentation/Splunk/8.2.6/Indexer/Takeapeeroffline#Take_a_peer_down_permane...

[discenzadoe]

What I meant by *nearly* everywhere is that there are some decommissioned server VMs that have been restarted (rarely), with a UF pointing to the old indexers. I don't have the rights to activate all of the old servers to make certain nothing still points to the indexers I wish to retire.

Additionally, the two indexers in question are *not* cluster members, so the command you listed would have zero effect on the standalone boxes.

Before the introduction of the indexer cluster, we had two indexers essentially load-balancing each other in distributed search, and those indexers are what I'm trying to retire.

[richgalloway · Accepted solution]

I read the OP as saying all five indexers are in a cluster.

Since you mention force-freezing data I presume you don't need to keep the data on these indexers.  Is that right?

If you don't want to keep the data then just remove the indexers from each SH's list of search peers then shut them down.

If you do want to keep the data then the buckets will have to be converted into cluster format and copied to the other indexers.  Then each clustered indexer will have to be restarted to import the new buckets.