Splunk Enterprise

Splunk Clustered Indexer data erasing

smithke
Explorer

We have three indexers in our cluster.

We want to decommission one of the indexers but still want to search all the data on it. Therefore we are moving the old historical data from the indexer we want to decommission to another indexer in the cluster.

We have migrated all indexes over to the other indexer, however, there are about 5 indexes that as soon as we copy the data over to the other indexer it gets wiped from that indexer. Again we copy the data from the indexer we want to decommission and again on the indexer we want to keep it is nearly immediately removed.

I am guessing this is a cluster setting that is removing it. What am I missing? We only have to get these remaining indexes off so we can decommission this indexer.

Let me know if you need more clarification on the issue. I also have opened a ticket with Splunk support on the issue and will post the resolution here if support finds the issue before the community does.

 

Labels (3)
Tags (1)
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You're trying too hard.  Just shut down the indexer with splunk offline --enforce-counts and let the Cluster Manager handle moving the data.

Also, make sure the cluster's RF/SF settings are consistent with a 2-node cluster.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

You're trying too hard.  Just shut down the indexer with splunk offline --enforce-counts and let the Cluster Manager handle moving the data.

Also, make sure the cluster's RF/SF settings are consistent with a 2-node cluster.

---
If this reply helps you, Karma would be appreciated.

smithke
Explorer

Will this copy over all data so that we ensure we dont lose anything?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes.  The --enforce-counts option tells the CM to make sure the RF and SF counts are satisfied before the indexer shuts down.

See https://docs.splunk.com/Documentation/Splunk/8.2.0/Indexer/Takeapeeroffline#Take_a_peer_down_permane... and https://docs.splunk.com/Documentation/Splunk/8.2.0/Indexer/Whathappenswhenapeergoesdown#When_a_peer_... .

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Admin Console: A Single, Unified Interface for All Your Cloud Admin Needs

WATCH NOWJoin us to learn how the admin console can save you time and give you more control over the Splunk® ...