How do I rebuilt a Cluster Master without a backup?

[garias_splunk]

Due to a disaster the Cluster Master of my indexer cluster is gone. There is no way to recover its data and we do not have a backup of its configuration files. 

The peers keeps working fine so far without the CM but we need to built a new instance from the scratch and configure it.

How can I do that?

[garias_splunk · Accepted solution]

A) If you have the backup of the host (server, files)- Here are the steps to replace the Cluster Master
https://docs.splunk.com/Documentation/Splunk/8.1.3/Indexer/Handlemanagernodefailure

OR

B) If you don't have the backup, you will need to do all configuration manually:

1 - You need to get the pass4SymmKey from any indexer. There may be hashed keys in there such as SSL passwords or the pass4SymmKey itself. You'll need to use CLI on the indexers to recover the original string
Use the command: splunk show-decrypted --value '<hashed_secret>'

2 - You need the contents of $SPLUNK_HOME/etc/slave_apps from one of the indexers. This will seed $SPLUNK_HOME/etc/master_apps on the new master

3 - IMPORTANT: make sure you harvest the contents of slave-apps from the indexer before you attempt to bring up any CM functionality on the new box; it starts with a blank bundle, and could overwrite the copy that the indexers have.

4- pass4simKey and the configuration bundle are pretty much the only state the CM maintains, so once you have those you should be good to go. the rest, the CM discovers from the indexers when they rejoin

5 - when setting up the server.conf,
[clustering] mode = manager pass4SymmKey = plaintext of pass4SymmKey search_factor = <search_factor> rep_factor = <rep_factor>

6 - So when you have the master_apps folder in the CM with the contents from slave-apps from the indexer , and all the settings , for example, server.conf, you can start and the CM will get the state of things from the indexers and decide what to do with terms of fixup.

7 - if there are any apps in slave-apps on the indexers that have encrypted pass4SymmKey values or other encrypted stuff like SSL Passwords or anything like that, you'll want to ensure that in master-apps on the CM, it's plaintext. That way it'll be pushed to the indexers and they will then encrypt it again using their own splunk.secret - the only caveat here being that if you have decided to use one single splunk.secret across your entire estate, you don't have to go with plaintext. You can just use the encrypted value you found on the indexer.