Deployment Architecture

What are the next steps to prep servers for retirement?

discenzadoe
Explorer

We have a distributed search environment, with 2 very old indexers (the original servers) and 3 new indexers in a cluster. 

The old indexers have been removed from the destination lists in outputs.conf nearly everywhere, and most of the data is between 5 and 6 months old, except for internal indexes.

I can't find what my next steps are to prep these servers for retirement, such as force-freezing the buckets they still hold, etc. 

Suggestions?

Thanks.

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

I read the OP as saying all five indexers are in a cluster.

Since you mention force-freezing data I presume you don't need to keep the data on these indexers.  Is that right?

If you don't want to keep the data then just remove the indexers from each SH's list of search peers then shut them down.

If you do want to keep the data then the buckets will have to be converted into cluster format and copied to the other indexers.  Then each clustered indexer will have to be restarted to import the new buckets.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The first step is to remove the old indexers from outputs.conf *everywhere*, not just nearly.

The next step is to run the command splunk offline --enforce-counts on one indexer.  This will tell the cluster to make sure the buckets on the old indexer exist elsewhere in the cluster.  Then the indexer will stop itself.

The last step is to repeat the previous step on the remaining indexer.

See https://docs.splunk.com/Documentation/Splunk/8.2.6/Indexer/Takeapeeroffline#Take_a_peer_down_permane...

---
If this reply helps you, Karma would be appreciated.
0 Karma

discenzadoe
Explorer

What I meant by *nearly* everywhere is that there are some decommissioned server VMs that have been restarted (rarely), with a UF pointing to the old indexers. I don't have the rights to activate all of the old servers to make certain nothing still points to the indexers I wish to retire.

Additionally, the two indexers in question are *not* cluster members, so the command you listed would have zero effect on the standalone boxes.

Before the introduction of the indexer cluster, we had two indexers essentially load-balancing each other in distributed search, and those indexers are what I'm trying to retire.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I read the OP as saying all five indexers are in a cluster.

Since you mention force-freezing data I presume you don't need to keep the data on these indexers.  Is that right?

If you don't want to keep the data then just remove the indexers from each SH's list of search peers then shut them down.

If you do want to keep the data then the buckets will have to be converted into cluster format and copied to the other indexers.  Then each clustered indexer will have to be restarted to import the new buckets.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...