Deployment Architecture

Why am I unable to add a new member to a search head cluster with constant "Your session is invalid. Please login." errors?

christian_l
Path Finder

Hi all,

I got a running search head cluster on latest Splunk Version 6.4.1.
I now need to add an additional member to the Cluster, and processed as described here:
http://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/Addaclustermember

I'm not struggling with the process Add_the_instance.
When using the command on the new member, I receive a login prompt asking over and over again for my username and password.
I checked the password and made sure it is correct. I double proofed it by entering a wrong password which directly exits the "authentication loop".

splunk4.my.net:/opt/splunk # /opt/splunk/bin/splunk add shcluster-member -current_member_uri https://splunk3.my.net:8089

Your session is invalid.  Please login.
Splunk username: admin
Password:
Your session is invalid.  Please login.
Splunk username: admin
Password:
Your session is invalid.  Please login.
Splunk username: admin
Password:
Your session is invalid.  Please login.
Splunk username: admin
Password:
[and so on and on ....]

Thankfully there is a second way, which allows adding a new member to the cluster. So I ran the following command on one of the members which are already within the cluster, which also ends with an error:

splunk3.my.net::/opt/splunk # /opt/splunk/bin/splunk add shcluster-member -new_member_uri https://splunk4.my.net:8089

In handler 'shclustermemberconsensus': Failed to Set Configuration. One potential reason is captain could not hear back from all the nodes in a timeout period. Ensure all to be added nodes are up, and increase the raft timeout. If all nodes are up and running, look at splunkd.log for appendEntries errors due to mgmt_uri mismatch

I saw the above error mentioned in other answers which point to a password mismatch. So I added the password in cleartext to the server.conf and also gave the already hashed pass4SymKey from another instance a chance. Situation doesn't change.
I also checked the network connectivity by telnet to port 8089 from the new system to a cluster member and vice versa - both directions are working.

How can I add the new node to the search head cluster?
Any ideas or recommendations?

Btw: I already restarted setting up the new node by removing and installing Splunk again for several times - every time the same result.

Greets
Christian

0 Karma

jmantor
Path Finder

I'm having the exact same problem on 6.3.6 in my QA environment.

0 Karma

sowings
Splunk Employee
Splunk Employee

Are the management passwords the same on both nodes? One thing that's not readily apparent (imo) is that when you use the "current_member_uri" method is that you have to provide the credentials of the remote node, not the local one.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...