I am trying to add a new member to a search head cluster.
Init the server with:
$SPLUNK_HOME/bin/splunk init shcluster-config -mgmt_uri https://server1:8089 -replication_port 8095 -replication_factor 3 -conf_deploy_fetch_url https://deployer:8088 -secret secretpw
Trying to add the server:
splunk add shcluster-member -current_member_uri https://server2:8089
Get prompted for username and password. Use admin and pw, but keeps prompting for username and password. Also tried a pw of changeme but then I get authentication failed.
I currently have 3 searchheads in the cluster.
You should check in splunkd and see what errors are being given. Most likely your secret password for the cluster is incorrect.
Also, is your deployer on 8088?
Yes deployer is 8088.
Reset passwords and am still seeing the same issues.
I completely stopped all searchheads and reinitialized the cluster. Brought everything back up and was able to get it working.
The reason this is happening is because the pass4SymmKey/shared-secret on your new member was not matching
the secret of the rest of the cluster.
When you ran the CLI command splunk add shcluster-member -currentmemberuri https://server2:8089,
an authentication token was generated on the search head server1 and put in its $HOME/.splunk directory.
add shcluster-member proxies the request to one of the current members (server 2) and tries to get the current
node added via the captain. This proxy process fails due to authentication, when the invalid token is used.
What you will need to do is.
1. bring down the node that is being added and fails. 2. go to server.conf of that node and change its pass4SymmKey to correct shared secret. or re init the node. 3. delete the $HOME/.splunk folder so that the CLI gets a new key 4. restart splunk 5. try the add shcluster-member command again.