Splunk Clustered Indexer data erasing

[smithke]

We have three indexers in our cluster.

We want to decommission one of the indexers but still want to search all the data on it. Therefore we are moving the old historical data from the indexer we want to decommission to another indexer in the cluster.

We have migrated all indexes over to the other indexer, however, there are about 5 indexes that as soon as we copy the data over to the other indexer it gets wiped from that indexer. Again we copy the data from the indexer we want to decommission and again on the indexer we want to keep it is nearly immediately removed.

I am guessing this is a cluster setting that is removing it. What am I missing? We only have to get these remaining indexes off so we can decommission this indexer.

Let me know if you need more clarification on the issue. I also have opened a ticket with Splunk support on the issue and will post the resolution here if support finds the issue before the community does.

[richgalloway · Accepted solution]

You're trying too hard.  Just shut down the indexer with splunk offline --enforce-counts and let the Cluster Manager handle moving the data.

Also, make sure the cluster's RF/SF settings are consistent with a 2-node cluster.

[smithke]

Will this copy over all data so that we ensure we dont lose anything?

[richgalloway]

Yes.  The --enforce-counts option tells the CM to make sure the RF and SF counts are satisfied before the indexer shuts down.

See https://docs.splunk.com/Documentation/Splunk/8.2.0/Indexer/Takeapeeroffline#Take_a_peer_down_permane... and https://docs.splunk.com/Documentation/Splunk/8.2.0/Indexer/Whathappenswhenapeergoesdown#When_a_peer_... .