Deployment Architecture

Why am I unable to add a new member to a search head cluster with constant "Your session is invalid. Please login." errors?

christian_l
Path Finder

Hi all,

I got a running search head cluster on latest Splunk Version 6.4.1.
I now need to add an additional member to the Cluster, and processed as described here:
http://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/Addaclustermember

I'm not struggling with the process Add_the_instance.
When using the command on the new member, I receive a login prompt asking over and over again for my username and password.
I checked the password and made sure it is correct. I double proofed it by entering a wrong password which directly exits the "authentication loop".

splunk4.my.net:/opt/splunk # /opt/splunk/bin/splunk add shcluster-member -current_member_uri https://splunk3.my.net:8089

Your session is invalid.  Please login.
Splunk username: admin
Password:
Your session is invalid.  Please login.
Splunk username: admin
Password:
Your session is invalid.  Please login.
Splunk username: admin
Password:
Your session is invalid.  Please login.
Splunk username: admin
Password:
[and so on and on ....]

Thankfully there is a second way, which allows adding a new member to the cluster. So I ran the following command on one of the members which are already within the cluster, which also ends with an error:

splunk3.my.net::/opt/splunk # /opt/splunk/bin/splunk add shcluster-member -new_member_uri https://splunk4.my.net:8089

In handler 'shclustermemberconsensus': Failed to Set Configuration. One potential reason is captain could not hear back from all the nodes in a timeout period. Ensure all to be added nodes are up, and increase the raft timeout. If all nodes are up and running, look at splunkd.log for appendEntries errors due to mgmt_uri mismatch

I saw the above error mentioned in other answers which point to a password mismatch. So I added the password in cleartext to the server.conf and also gave the already hashed pass4SymKey from another instance a chance. Situation doesn't change.
I also checked the network connectivity by telnet to port 8089 from the new system to a cluster member and vice versa - both directions are working.

How can I add the new node to the search head cluster?
Any ideas or recommendations?

Btw: I already restarted setting up the new node by removing and installing Splunk again for several times - every time the same result.

Greets
Christian

0 Karma

jmantor
Path Finder

I'm having the exact same problem on 6.3.6 in my QA environment.

0 Karma

sowings
Splunk Employee
Splunk Employee

Are the management passwords the same on both nodes? One thing that's not readily apparent (imo) is that when you use the "current_member_uri" method is that you have to provide the credentials of the remote node, not the local one.

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...