Deployment Architecture

Why do I keep getting prompted for a username and password trying to add a member to a search head cluster?

rmorlen
Splunk Employee
Splunk Employee

I am trying to add a new member to a search head cluster.

Init the server with:

$SPLUNK_HOME/bin/splunk init shcluster-config -mgmt_uri https://server1:8089 -replication_port 8095 -replication_factor 3 -conf_deploy_fetch_url https://deployer:8088 -secret secretpw

Trying to add the server:

splunk add shcluster-member -current_member_uri https://server2:8089

Get prompted for username and password. Use admin and pw, but keeps prompting for username and password. Also tried a pw of changeme but then I get authentication failed.

I currently have 3 searchheads in the cluster.

0 Karma
1 Solution

rmorlen
Splunk Employee
Splunk Employee

I completely stopped all searchheads and reinitialized the cluster. Brought everything back up and was able to get it working.

View solution in original post

0 Karma

arahut_splunk
Splunk Employee
Splunk Employee

The reason this is happening is because the pass4SymmKey/shared-secret on your new member was not matching
the secret of the rest of the cluster.

When you ran the CLI command splunk add shcluster-member -current_member_uri https://server2:8089,
an authentication token was generated on the search head server1 and put in its $HOME/.splunk directory.

add shcluster-member proxies the request to one of the current members (server 2) and tries to get the current
node added via the captain. This proxy process fails due to authentication, when the invalid token is used.

What you will need to do is.

 1. bring down the node that is being added and fails.
 2. go to server.conf of that node and change its pass4SymmKey to correct shared secret.   or re  init the node.
 3. delete the $HOME/.splunk folder so that the CLI gets a new key
 4. restart splunk
 5. try the add shcluster-member command again.

rmorlen
Splunk Employee
Splunk Employee

I completely stopped all searchheads and reinitialized the cluster. Brought everything back up and was able to get it working.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

You should check in splunkd and see what errors are being given. Most likely your secret password for the cluster is incorrect.

Also, is your deployer on 8088?

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

What is the message in splunkd.log say?

0 Karma

rmorlen
Splunk Employee
Splunk Employee

Yes deployer is 8088.

Reset passwords and am still seeing the same issues.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...