Installation

Implications of Destroying an Indexer Peer Within a Cluster

TheColorBlack
Path Finder

Evening Splunk Community,

I'd like to better understand the consequences of destroying a single indexer peer within my indexer cluster. To make a long story short, while resizing the root partition on one of my indexers I managed to mangle the partition, and the system will no longer boot.

Prior to mangling the effected indexer I did offline the peer by executing temporary indexer shutdown command below.

 

 

splunk offline

 

 

 
Once it was evident I wasn't going to be able to save the affected partition, I decided to build a new indexer, remove the mangled indexer from my cluster, and join the new replacement indexer into the cluster.

I removed the affected indexer from my cluster by executing:

 

 

splunk remove cluster-peers -peers <guid>

 

 

 

What I would like to understand is if I've managed to destroy any data in my cluster, or the next steps I need to take to bring my cluster back up to full speed.

My cluster consists of six indexing peers with a replication factor of 3, and a search factor of 2. Which leads me to believe that my other indexers contain a replica copy of the data I potentially destroyed on the affected indexer I managed to mangle. Is this true?

I believe the only thing left to do now is to perform a data re-balance to equalize the storage utilization across my indexer peers. 

Labels (1)
0 Karma
1 Solution
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!