Deployment Architecture

Splunk indexer node unable to join the cluster

skrish91
Path Finder

I have an indexer cluster with 1 master and 2 peer nodes. The peer nodes machine got rebooted suddenly and now 1 of the peer node is showing status DOWN. I have tried restarting the node but it doesnt help. The replication factor for the cluster is 2. I have also tried adding the node to the cluster again and it complains the secret key is wrong. Is there any way I could find the secret key from the master or the working node? What is the best way to fix this issue and make the peer join the cluster?

0 Karma
1 Solution

nareshinsvu
Builder

As everyone mentioned, change the server.conf files on Master and indexer nodes
followed by a splunk restart of Master and then indexers

https://docs.splunk.com/Documentation/Splunk/7.2.6/Indexer/Enableclustersindetail#Configure_the_secu...

View solution in original post

0 Karma

nareshinsvu
Builder

As everyone mentioned, change the server.conf files on Master and indexer nodes
followed by a splunk restart of Master and then indexers

https://docs.splunk.com/Documentation/Splunk/7.2.6/Indexer/Enableclustersindetail#Configure_the_secu...

View solution in original post

0 Karma

Vijeta
Influencer

@skrish91 if you do not have access to keys, then assign new keys . It must be pass4symmkey in server.conf for cluster.
Assign a new password/key on all the 3 servers and rebooting would encrypt it. Make sure you change pass4symmkey only for your indexer cluster stanza in server.conf.

Use this document-

https://docs.splunk.com/Documentation/Splunk/7.2.6/Indexer/Enableclustersindetail#Configure_the_secu...

0 Karma

skrish91
Path Finder

I tried changing the pass4symmkey under [clustering] section. When I change that and restart the services on all the machines, master and node2 is ok but the node1 shows its DOWN. This was weird.

0 Karma

Jarohnimo
Builder

Change the pass4 key to whatever you'd like. You'd have to change it in both cluster master and indexer nodes if you don't remember It, and then restart the nodes starting with the Master node first (you can do it from the gui).

0 Karma

codebuilder
Influencer

The pass4SymmKey under [general] also needs to match on all nodes in the cluster.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!