Retiring indexers

[BrianAbbott]

I have an older cluster that is currently aging out. The peer group consists of three indexers which need to be dropped down to just one. RF=3 and SF=1 at this time.

I want to be sure that I am taking the correct steps to remove two of these indexers would be. Looking at https://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Removepeerfrommasterlist it seems that I need only:

splunk remove cluster-peers -peers <guid>

Is that correct?

Will the SF be an issue or will it rebuild automatically?

Thanks.

[BrianAbbott]

I did set the RF to 1 (SF by default) because it esdbeen my understanding that the RF has to be set to 1 in order to accomplish what I am trying to do. Now that I reread it, perhaps I need(ed) to set the RF to be 2, as it had been 3.

https://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Takeapeeroffline#Take_a_peer_down_permane...
Because this version of splunk offline requires that the cluster return to a complete state before the peer can go down, certain preconditions are necessary before you can run this command:
• The cluster must have (replication factor + 1) number of peers, so that it can reallocate bucket copies to other peers as necessary and can continue to meet its replication factor after the peer goes down.

[BrianAbbott]

I appreciate the follow up, quite helpful.

I did begin the process by putting one peer into a decommissioning status, where it has remained even until now. RF=1 SF=1. RF is met but one index is being a pain. It seems that one index has a bucket with no possible primaries. As such, I could delete the copy.

Is it acceptable to place the next indexer into decommissioning status even as the SF is technically not met?

[tiagofbmm]

No, you need to wait for the decommission to finish. Let it keep running for longer and that should be fixed

[tiagofbmm]

But wait, you changed the RF and SF before finishing the decommission?
Every bucket needs to have a primary. That's the premise for you to have all your data searchable
If your SF is 1 and it is not met, you'd need to later rebuild the tsidx files for that bucket later from the raw data (RF data)

[tiagofbmm]

Go with splunk offline --enforce-counts. Used to remove a peer permanently from the cluster. Also known as the "enforce-counts offline" command.

http://docs.splunk.com/Documentation/Splunk/6.6.3
/Indexer/Takeapeeroffline.

it will get you the cluster in a valid state on the process. Remember you're moving to 1 Indexer only, you need to change the RF to 1 otherwise it won't be valid after you finish the decommission. SF will be fine as long as you bring the peers offline one at a time as the buckets will be copied to the last one standing, no need to rebuild or make them searchable again

After that you can remove the peer from the cluster as you mentioned