Installation

Implications of Destroying an Indexer Peer Within a Cluster

TheColorBlack
Path Finder

Evening Splunk Community,

I'd like to better understand the consequences of destroying a single indexer peer within my indexer cluster. To make a long story short, while resizing the root partition on one of my indexers I managed to mangle the partition, and the system will no longer boot.

Prior to mangling the effected indexer I did offline the peer by executing temporary indexer shutdown command below.

 

 

splunk offline

 

 

 
Once it was evident I wasn't going to be able to save the affected partition, I decided to build a new indexer, remove the mangled indexer from my cluster, and join the new replacement indexer into the cluster.

I removed the affected indexer from my cluster by executing:

 

 

splunk remove cluster-peers -peers <guid>

 

 

 

What I would like to understand is if I've managed to destroy any data in my cluster, or the next steps I need to take to bring my cluster back up to full speed.

My cluster consists of six indexing peers with a replication factor of 3, and a search factor of 2. Which leads me to believe that my other indexers contain a replica copy of the data I potentially destroyed on the affected indexer I managed to mangle. Is this true?

I believe the only thing left to do now is to perform a data re-balance to equalize the storage utilization across my indexer peers. 

Labels (1)
0 Karma
1 Solution
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...