Deployment Architecture

What are the next steps to prep servers for retirement?

discenzadoe
Explorer

We have a distributed search environment, with 2 very old indexers (the original servers) and 3 new indexers in a cluster. 

The old indexers have been removed from the destination lists in outputs.conf nearly everywhere, and most of the data is between 5 and 6 months old, except for internal indexes.

I can't find what my next steps are to prep these servers for retirement, such as force-freezing the buckets they still hold, etc. 

Suggestions?

Thanks.

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

I read the OP as saying all five indexers are in a cluster.

Since you mention force-freezing data I presume you don't need to keep the data on these indexers.  Is that right?

If you don't want to keep the data then just remove the indexers from each SH's list of search peers then shut them down.

If you do want to keep the data then the buckets will have to be converted into cluster format and copied to the other indexers.  Then each clustered indexer will have to be restarted to import the new buckets.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The first step is to remove the old indexers from outputs.conf *everywhere*, not just nearly.

The next step is to run the command splunk offline --enforce-counts on one indexer.  This will tell the cluster to make sure the buckets on the old indexer exist elsewhere in the cluster.  Then the indexer will stop itself.

The last step is to repeat the previous step on the remaining indexer.

See https://docs.splunk.com/Documentation/Splunk/8.2.6/Indexer/Takeapeeroffline#Take_a_peer_down_permane...

---
If this reply helps you, Karma would be appreciated.
0 Karma

discenzadoe
Explorer

What I meant by *nearly* everywhere is that there are some decommissioned server VMs that have been restarted (rarely), with a UF pointing to the old indexers. I don't have the rights to activate all of the old servers to make certain nothing still points to the indexers I wish to retire.

Additionally, the two indexers in question are *not* cluster members, so the command you listed would have zero effect on the standalone boxes.

Before the introduction of the indexer cluster, we had two indexers essentially load-balancing each other in distributed search, and those indexers are what I'm trying to retire.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I read the OP as saying all five indexers are in a cluster.

Since you mention force-freezing data I presume you don't need to keep the data on these indexers.  Is that right?

If you don't want to keep the data then just remove the indexers from each SH's list of search peers then shut them down.

If you do want to keep the data then the buckets will have to be converted into cluster format and copied to the other indexers.  Then each clustered indexer will have to be restarted to import the new buckets.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Want a chance to win $500 to the Splunk shop? Take our IT Incident Management Survey!

  Top Trends & Best Practices in Incident ManagementSplunk is partnering up with Constellation Research to ...