Deployment Architecture

Why do I keep getting prompted for a username and password trying to add a member to a search head cluster?

rmorlen
Splunk Employee
Splunk Employee

I am trying to add a new member to a search head cluster.

Init the server with:

$SPLUNK_HOME/bin/splunk init shcluster-config -mgmt_uri https://server1:8089 -replication_port 8095 -replication_factor 3 -conf_deploy_fetch_url https://deployer:8088 -secret secretpw

Trying to add the server:

splunk add shcluster-member -current_member_uri https://server2:8089

Get prompted for username and password. Use admin and pw, but keeps prompting for username and password. Also tried a pw of changeme but then I get authentication failed.

I currently have 3 searchheads in the cluster.

0 Karma
1 Solution

rmorlen
Splunk Employee
Splunk Employee

I completely stopped all searchheads and reinitialized the cluster. Brought everything back up and was able to get it working.

View solution in original post

0 Karma

arahut_splunk
Splunk Employee
Splunk Employee

The reason this is happening is because the pass4SymmKey/shared-secret on your new member was not matching
the secret of the rest of the cluster.

When you ran the CLI command splunk add shcluster-member -current_member_uri https://server2:8089,
an authentication token was generated on the search head server1 and put in its $HOME/.splunk directory.

add shcluster-member proxies the request to one of the current members (server 2) and tries to get the current
node added via the captain. This proxy process fails due to authentication, when the invalid token is used.

What you will need to do is.

 1. bring down the node that is being added and fails.
 2. go to server.conf of that node and change its pass4SymmKey to correct shared secret.   or re  init the node.
 3. delete the $HOME/.splunk folder so that the CLI gets a new key
 4. restart splunk
 5. try the add shcluster-member command again.

rmorlen
Splunk Employee
Splunk Employee

I completely stopped all searchheads and reinitialized the cluster. Brought everything back up and was able to get it working.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

You should check in splunkd and see what errors are being given. Most likely your secret password for the cluster is incorrect.

Also, is your deployer on 8088?

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

What is the message in splunkd.log say?

0 Karma

rmorlen
Splunk Employee
Splunk Employee

Yes deployer is 8088.

Reset passwords and am still seeing the same issues.

0 Karma
Get Updates on the Splunk Community!

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...