Why we are unable to add a cluster member via CLI to our existing search head cluster?

[att35]

Hi,

We created a new Search Head Cluster that includes one Deployer and 2 Cluster members with one being the captain. Deployment went well and the cluster members can recognize each other.

 Captain:
                          dynamic_captain : 1
                          elected_captain : Thu May 12 15:37:23 2016
                                       id : *******************************
                         initialized_flag : 1
                                    label : splunk03.x.y.z
                         maintenance_mode : 0
                                 mgmt_uri : https://splunk03.x.y.z:8089
                    min_peers_joined_flag : 1
                     rolling_restart_flag : 0
                       service_ready_flag : 1

 Members:
        splunk03.x.y.z
                                    label : splunk03.x.y.z
                                 mgmt_uri : https://splunk03.x.y.z:8089
                           mgmt_uri_alias : https://X.X.X.56:8089
                                   status : Up
        splunk04.x.y.z
                                    label : splunk04.x.y.z
                                 mgmt_uri : https://splunk04.x.y.z:8089
                           mgmt_uri_alias : https://X.X.X.57:8089
                                   status : Up

But now when we are trying to add another member, it is giving errors. We tried both options. From an existing member using splunk add shcluster-member -new_member_uri <URI>:<management_port> and from the new member, using splunk add shcluster-member -current_member_uri <URI>:<management_port>

While trying from the new member, packet capture shows communication between splunk05 (New member) and splunk04 (existing member). In splunkd.log on splunk05, following messages are repeated.

05-20-2016 16:04:59.958 -0400 WARN  SHClusterHandler - Failed to proxy call to member. https://splunk04.x.y.z:8089 WARN:  call not properly authenticated
05-20-2016 16:05:00.081 -0400 WARN  SHClusterHandler - Failed to proxy call to member. https://splunk04.x.y.z:8089 WARN:  call not properly authenticated

• Verified server.conf for all members and made sure mgmt_uri is correct.
• All members have same value for replication_factor, replication_port, shcluster_lable and pass4SymmKey.
• Firewall rules allow communication on management port.
• Admin credential being used to authenticate are correct.

I could not find any articles referring to this proxy error. Are we missing anything obvious? Are these only warnings which can be ignored?

Thanks in advance..

~ Abhi

[alibrahim]

I was getting this error because the time was not synchronized on all of the search head cluster members:

09-02-2016 12:37:45.549 -0400 WARN  SessionManager - Rejecting expired token generated by KR619C8B-C9D3-BB80-40FC-5F8574404AD4 because its expiration time 1468596424 is earlier than the current time 1472834265

[dolivasoh · Accepted solution]

The secret used to hash your pass4SymmKey on the new search head is most likely different than the others. Put this value in plain text and restart to have it encrypted to the proper value.

[att35]

Thank you. That was it..

Added the new key in plaintext and restarted that Splunk instance. Was able to add it as a member successfully.

[rashi83]

pass4SymmKey is same on all nodes and deployer. I am able to create a captain and unable to add any member to the cluster.
What could be the issue?

[root@usdf24v0119 bin]# ./splunk show shcluster-status -auth admin:changeme

Captain:
dynamic_captain : 1
elected_captain : Thu Oct 18 13:16:00 2018
id : A4EA45C1-9811-492F-B45C-AA22C40D7E8B
initialized_flag : 0
label : usdf24v0119
mgmt_uri : http://usdf24v0119:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0

Members:
usdf24v0119
label : usdf24v0119
mgmt_uri : http://usdf24v0119:8089
mgmt_uri_alias : https://10.23.132.155:8089
status : Up