Deployment Architecture
Highlighted

Why do I keep getting prompted for a username and password trying to add a member to a search head cluster?

Splunk Employee
Splunk Employee

I am trying to add a new member to a search head cluster.

Init the server with:

$SPLUNK_HOME/bin/splunk init shcluster-config -mgmt_uri https://server1:8089 -replication_port 8095 -replication_factor 3 -conf_deploy_fetch_url https://deployer:8088 -secret secretpw

Trying to add the server:

splunk add shcluster-member -current_member_uri https://server2:8089

Get prompted for username and password. Use admin and pw, but keeps prompting for username and password. Also tried a pw of changeme but then I get authentication failed.

I currently have 3 searchheads in the cluster.

0 Karma
Highlighted

Re: Why do I keep getting prompted for a username and password trying to add a member to a search head cluster?

Splunk Employee
Splunk Employee

You should check in splunkd and see what errors are being given. Most likely your secret password for the cluster is incorrect.

Also, is your deployer on 8088?

0 Karma
Highlighted

Re: Why do I keep getting prompted for a username and password trying to add a member to a search head cluster?

Splunk Employee
Splunk Employee

Yes deployer is 8088.

Reset passwords and am still seeing the same issues.

0 Karma
Highlighted

Re: Why do I keep getting prompted for a username and password trying to add a member to a search head cluster?

Splunk Employee
Splunk Employee

What is the message in splunkd.log say?

0 Karma
Highlighted

Re: Why do I keep getting prompted for a username and password trying to add a member to a search head cluster?

Splunk Employee
Splunk Employee

I completely stopped all searchheads and reinitialized the cluster. Brought everything back up and was able to get it working.

View solution in original post

0 Karma
Highlighted

Re: Why do I keep getting prompted for a username and password trying to add a member to a search head cluster?

Splunk Employee
Splunk Employee

The reason this is happening is because the pass4SymmKey/shared-secret on your new member was not matching
the secret of the rest of the cluster.

When you ran the CLI command splunk add shcluster-member -currentmemberuri https://server2:8089,
an authentication token was generated on the search head server1 and put in its $HOME/.splunk directory.

add shcluster-member proxies the request to one of the current members (server 2) and tries to get the current
node added via the captain. This proxy process fails due to authentication, when the invalid token is used.

What you will need to do is.

 1. bring down the node that is being added and fails.
 2. go to server.conf of that node and change its pass4SymmKey to correct shared secret.   or re  init the node.
 3. delete the $HOME/.splunk folder so that the CLI gets a new key
 4. restart splunk
 5. try the add shcluster-member command again.