How to troubleshoot why a Splunk peer node is failing to connect to the master node in an indexer clustering environment?

[amoldesai]

I am using Splunk 6.3.3. My Splunk indexer clustering environment was working fine earlier. Now one of the peer nodes is unable to connect to the master node.

I don't see the peer node in the indexer view of the master dashboard. When I try to view the Setting-->Distributed Environment-->Index Clustering view on the peer node, I get the following error message:

In handler 'clusterslavecontrol': Slave is unable to handle request at this time.  This means either slave unable to communicate w/ master OR master does not have  peers added to the cluster.  Check master dashboard and/or master_uri/secret settings.

Request your help in troubleshooting this problem.

Thanks

[ddrillic]

http://docs.splunk.com/Documentation/Splunk/6.3.0/RESTREF/RESTcluster speaks about it -

It says about secret -
Secret shared among the nodes in the cluster to prevent any arbitrary node from connecting to the cluster. If a peer or searchhead is not configured with the same secret as the master, it is not able to communicate with the master.

[amoldesai]

Thanks for your reply.

I now have the same secret key as that in the master node(pass4SymmKey value in the server.conf file under local directory).
But I continue to get the same error.
Any ideas ?

As this is my dev environment, I am also ok to clean the data/configuration on this peer node and reconfigure it. What steps need to followed to reconfigure the peer node.

Thanks

[amoldesai]

More information regarding the above issue. In the UI of master mode, I see a message as :

Failed to add peer 'guid=E975B264-926A-4E16-9FC9-4AE355086032 server name= ip=:8089 timed out; exceeded 30sec.

Thanks