Deployment Architecture

What are the next steps to prep servers for retirement?

discenzadoe
Explorer

We have a distributed search environment, with 2 very old indexers (the original servers) and 3 new indexers in a cluster. 

The old indexers have been removed from the destination lists in outputs.conf nearly everywhere, and most of the data is between 5 and 6 months old, except for internal indexes.

I can't find what my next steps are to prep these servers for retirement, such as force-freezing the buckets they still hold, etc. 

Suggestions?

Thanks.

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

I read the OP as saying all five indexers are in a cluster.

Since you mention force-freezing data I presume you don't need to keep the data on these indexers.  Is that right?

If you don't want to keep the data then just remove the indexers from each SH's list of search peers then shut them down.

If you do want to keep the data then the buckets will have to be converted into cluster format and copied to the other indexers.  Then each clustered indexer will have to be restarted to import the new buckets.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The first step is to remove the old indexers from outputs.conf *everywhere*, not just nearly.

The next step is to run the command splunk offline --enforce-counts on one indexer.  This will tell the cluster to make sure the buckets on the old indexer exist elsewhere in the cluster.  Then the indexer will stop itself.

The last step is to repeat the previous step on the remaining indexer.

See https://docs.splunk.com/Documentation/Splunk/8.2.6/Indexer/Takeapeeroffline#Take_a_peer_down_permane...

---
If this reply helps you, an upvote would be appreciated.
0 Karma

discenzadoe
Explorer

What I meant by *nearly* everywhere is that there are some decommissioned server VMs that have been restarted (rarely), with a UF pointing to the old indexers. I don't have the rights to activate all of the old servers to make certain nothing still points to the indexers I wish to retire.

Additionally, the two indexers in question are *not* cluster members, so the command you listed would have zero effect on the standalone boxes.

Before the introduction of the indexer cluster, we had two indexers essentially load-balancing each other in distributed search, and those indexers are what I'm trying to retire.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I read the OP as saying all five indexers are in a cluster.

Since you mention force-freezing data I presume you don't need to keep the data on these indexers.  Is that right?

If you don't want to keep the data then just remove the indexers from each SH's list of search peers then shut them down.

If you do want to keep the data then the buckets will have to be converted into cluster format and copied to the other indexers.  Then each clustered indexer will have to be restarted to import the new buckets.

---
If this reply helps you, an upvote would be appreciated.
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...