Installation

Implications of Destroying an Indexer Peer Within a Cluster

TheColorBlack
Path Finder

Evening Splunk Community,

I'd like to better understand the consequences of destroying a single indexer peer within my indexer cluster. To make a long story short, while resizing the root partition on one of my indexers I managed to mangle the partition, and the system will no longer boot.

Prior to mangling the effected indexer I did offline the peer by executing temporary indexer shutdown command below.

 

 

splunk offline

 

 

 
Once it was evident I wasn't going to be able to save the affected partition, I decided to build a new indexer, remove the mangled indexer from my cluster, and join the new replacement indexer into the cluster.

I removed the affected indexer from my cluster by executing:

 

 

splunk remove cluster-peers -peers <guid>

 

 

 

What I would like to understand is if I've managed to destroy any data in my cluster, or the next steps I need to take to bring my cluster back up to full speed.

My cluster consists of six indexing peers with a replication factor of 3, and a search factor of 2. Which leads me to believe that my other indexers contain a replica copy of the data I potentially destroyed on the affected indexer I managed to mangle. Is this true?

I believe the only thing left to do now is to perform a data re-balance to equalize the storage utilization across my indexer peers. 

Labels (1)
0 Karma
1 Solution
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...