How to move data from one peer node to another peer in an indexer clustering environment?

[dpraveen88]

I have 3 indexers in cluster master. (Indexer 1, indexer2 and indexer3)
I need to stop indexer2 and indexer3 permanently.
To Stop the indexers in the cluster, I use "offline" command. It stops the indexer offline. Now I need to move the buckets (data) from indexer2 ,3 to indexer 1.

Please help me the process steps to move buckets to existing indexers.

Thanks!

[lguinn2 · Accepted solution]

NO!! You cannot simply move buckets from one indexer to another. You shouldn't do this in general and you definitely shouldn't do it on an indexer cluster. Chances are high that you will corrupt all of the data. But there is a way to have Splunk do this for you, and it is pretty simple. First question: did you use

splunk offline --enforce-counts

or just

splunk offline

to take indexers offline? If you used "enforce-counts" AND you waited for each indexer to fully stop, then: congratulations! You are done!! The cluster master automatically made copies of all necessary data to the surviving indexer.

First, without enforce-counts, "offline" only means that the indexer is only going to be offline for a restart. That is not the case here. So, start the indexers (indexer2 and indexer3) again, and do the offline with enforce-counts. It may take a long time for each indexer to fully stop, but you need to wait it out.

[dpraveen88]

Thanks for responding for you valuable suggestion. I used already this command "splunk offline --enforce-counts". so far i stopped the indexer3 permanently. After that whatever the old data is available in indexer3, i need to move from indexer3 to indexer1.

[masonmorales]

If you have the storage, just increase the search factor and replication factor to 3 on the cluster master and let the buckets replicate on their own. Then, you can just remove indexer 2 and indexer 3 from the cluster.

[hunderliggur]

To reduce a cluster from 3 to 1:
Set search factor 1 replication factor 2.

Let the cluster stabilize.

Remove indexer 3 with a controlled stop to remove it from the cluster (splunk offline --enforce-counts).
Let the cluster stabilize.
Remove indexer 2 with a controlled stop to remove it from the cluster (splunk offline --enforce-counts).
Let the cluster stabilize.
You are all done.