Splunk Enterprise

Splunk Clustered Indexer data erasing

smithke
Explorer

We have three indexers in our cluster.

We want to decommission one of the indexers but still want to search all the data on it. Therefore we are moving the old historical data from the indexer we want to decommission to another indexer in the cluster.

We have migrated all indexes over to the other indexer, however, there are about 5 indexes that as soon as we copy the data over to the other indexer it gets wiped from that indexer. Again we copy the data from the indexer we want to decommission and again on the indexer we want to keep it is nearly immediately removed.

I am guessing this is a cluster setting that is removing it. What am I missing? We only have to get these remaining indexes off so we can decommission this indexer.

Let me know if you need more clarification on the issue. I also have opened a ticket with Splunk support on the issue and will post the resolution here if support finds the issue before the community does.

 

Labels (3)
Tags (1)
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You're trying too hard.  Just shut down the indexer with splunk offline --enforce-counts and let the Cluster Manager handle moving the data.

Also, make sure the cluster's RF/SF settings are consistent with a 2-node cluster.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

You're trying too hard.  Just shut down the indexer with splunk offline --enforce-counts and let the Cluster Manager handle moving the data.

Also, make sure the cluster's RF/SF settings are consistent with a 2-node cluster.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

smithke
Explorer

Will this copy over all data so that we ensure we dont lose anything?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes.  The --enforce-counts option tells the CM to make sure the RF and SF counts are satisfied before the indexer shuts down.

See https://docs.splunk.com/Documentation/Splunk/8.2.0/Indexer/Takeapeeroffline#Take_a_peer_down_permane... and https://docs.splunk.com/Documentation/Splunk/8.2.0/Indexer/Whathappenswhenapeergoesdown#When_a_peer_... .

---
If this reply helps you, an upvote would be appreciated.
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.