Evening Splunk Community,
I'd like to better understand the consequences of destroying a single indexer peer within my indexer cluster. To make a long story short, while resizing the root partition on one of my indexers I managed to mangle the partition, and the system will no longer boot.
Prior to mangling the effected indexer I did offline the peer by executing temporary indexer shutdown command below.
splunk offline
Once it was evident I wasn't going to be able to save the affected partition, I decided to build a new indexer, remove the mangled indexer from my cluster, and join the new replacement indexer into the cluster.
I removed the affected indexer from my cluster by executing:
splunk remove cluster-peers -peers <guid>
What I would like to understand is if I've managed to destroy any data in my cluster, or the next steps I need to take to bring my cluster back up to full speed.
My cluster consists of six indexing peers with a replication factor of 3, and a search factor of 2. Which leads me to believe that my other indexers contain a replica copy of the data I potentially destroyed on the affected indexer I managed to mangle. Is this true?
I believe the only thing left to do now is to perform a data re-balance to equalize the storage utilization across my indexer peers.
