Why am I unable to add a new member to a search head cluster with constant "Your session is invalid. Please login." errors?

[christian_l]

Hi all,

I got a running search head cluster on latest Splunk Version 6.4.1.
I now need to add an additional member to the Cluster, and processed as described here:
http://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/Addaclustermember

I'm not struggling with the process Add_the_instance.
When using the command on the new member, I receive a login prompt asking over and over again for my username and password.
I checked the password and made sure it is correct. I double proofed it by entering a wrong password which directly exits the "authentication loop".

splunk4.my.net:/opt/splunk # /opt/splunk/bin/splunk add shcluster-member -current_member_uri https://splunk3.my.net:8089

Your session is invalid.  Please login.
Splunk username: admin
Password:
Your session is invalid.  Please login.
Splunk username: admin
Password:
Your session is invalid.  Please login.
Splunk username: admin
Password:
Your session is invalid.  Please login.
Splunk username: admin
Password:
[and so on and on ....]

Thankfully there is a second way, which allows adding a new member to the cluster. So I ran the following command on one of the members which are already within the cluster, which also ends with an error:

splunk3.my.net::/opt/splunk # /opt/splunk/bin/splunk add shcluster-member -new_member_uri https://splunk4.my.net:8089

In handler 'shclustermemberconsensus': Failed to Set Configuration. One potential reason is captain could not hear back from all the nodes in a timeout period. Ensure all to be added nodes are up, and increase the raft timeout. If all nodes are up and running, look at splunkd.log for appendEntries errors due to mgmt_uri mismatch

I saw the above error mentioned in other answers which point to a password mismatch. So I added the password in cleartext to the server.conf and also gave the already hashed pass4SymKey from another instance a chance. Situation doesn't change.
I also checked the network connectivity by telnet to port 8089 from the new system to a cluster member and vice versa - both directions are working.

How can I add the new node to the search head cluster?
Any ideas or recommendations?

Btw: I already restarted setting up the new node by removing and installing Splunk again for several times - every time the same result.

Greets
Christian

[jmantor]

I'm having the exact same problem on 6.3.6 in my QA environment.

[sowings]

Are the management passwords the same on both nodes? One thing that's not readily apparent (imo) is that when you use the "current_member_uri" method is that you have to provide the credentials of the remote node, not the local one.