I have an indexer cluster with 1 master and 2 peer nodes. The peer nodes machine got rebooted suddenly and now 1 of the peer node is showing status DOWN. I have tried restarting the node but it doesnt help. The replication factor for the cluster is 2. I have also tried adding the node to the cluster again and it complains the secret key is wrong. Is there any way I could find the secret key from the master or the working node? What is the best way to fix this issue and make the peer join the cluster?
As everyone mentioned, change the server.conf files on Master and indexer nodes
followed by a splunk restart of Master and then indexers
As everyone mentioned, change the server.conf files on Master and indexer nodes
followed by a splunk restart of Master and then indexers
@skrish91 if you do not have access to keys, then assign new keys . It must be pass4symmkey in server.conf for cluster.
Assign a new password/key on all the 3 servers and rebooting would encrypt it. Make sure you change pass4symmkey only for your indexer cluster stanza in server.conf.
Use this document-
I tried changing the pass4symmkey under [clustering] section. When I change that and restart the services on all the machines, master and node2 is ok but the node1 shows its DOWN. This was weird.
Change the pass4 key to whatever you'd like. You'd have to change it in both cluster master and indexer nodes if you don't remember It, and then restart the nodes starting with the Master node first (you can do it from the gui).
The pass4SymmKey under [general] also needs to match on all nodes in the cluster.