Search Head Cluster - can't add members after captain bootstrap (8.1.2)?

[whar_garbl]

I am rebuilding a SH cluster from scratch. I've followed the documentation carefully to this point. I have the shcluster captain bootstrapped and splunk show shcluster-status shows the captain as the only member, but the bootstrapping process failed to add my member nodes due to comms errors. Pretty sure I've got those fixed now. 

When I do splunk add shcluster-member -current_member_uri https://ip-address-of-captain:8089 on a member node, it tells me: 

current_member_uri is pointing back to this same node. It should point to a node that is already a member of a cluster. 

Obviously, I have checked and re-checked the uri, which I believe is correct (https://ip-address-of-captain:8089), and that is set right in server.conf on both sides. There is no IP conflict and the servers have no issue communicating. 

If I run splunk add shcluster-member -new_member_uri https://ip-address-of-member:8089 from the captain, it tells me:

Failed to proxy call to member https://ip-address-of-member:8089

Google tells me this can be an issue with the pass4SymmKey, and to that end, I have updated the pass4SymmKey on both sides and restarted the instances a few times, to no avail. 

I'm stumped. Where did I go wrong that I can't get these search heads to cluster up nicely?

[loganac]

I had this exact issue today and here's what I did:

For my issue, the SHC had a static captain. So I followed the Splunk docs to try and get them to become a RAFT distributed consensus voting for the captain. When I ran the commands the SHC cluster broke. After looking around for a while in the conf files I change two things on the non-captain servers.

In server.conf, the mgmt_uri was pointing to the existing captain. That has to be its own self per instructions in server.conf and delete the captain_url stanza. After I deleted those I restarted Splunk and ran the command pointed to the captain who was still the cluster

splunk add shcluster-member -current_member_uri <URI>:<management_port>

I repeated that for the other hosts until the captain was left

When I went to the captain I made sure that "mode = member" and deleted the captain_url stanza. When I restarted that host was no longer the captain and another had picked it up.

Hope this helps