Deployment Architecture

Search Head Cluster - can't add members after captain bootstrap (8.1.2)?

whar_garbl
Path Finder

I am rebuilding a SH cluster from scratch. I've followed the documentation carefully to this point. I have the shcluster captain bootstrapped and splunk show shcluster-status shows the captain as the only member, but the bootstrapping process failed to add my member nodes due to comms errors. Pretty sure I've got those fixed now. 

When I do splunk add shcluster-member -current_member_uri https://ip-address-of-captain:8089 on a member node, it tells me: 

 

 

current_member_uri is pointing back to this same node. It should point to a node that is already a member of a cluster. 

 

 

Obviously, I have checked and re-checked the uri, which I believe is correct (https://ip-address-of-captain:8089), and that is set right in server.conf on both sides. There is no IP conflict and the servers have no issue communicating. 

If I run splunk add shcluster-member -new_member_uri https://ip-address-of-member:8089 from the captain, it tells me:

 

 

Failed to proxy call to member https://ip-address-of-member:8089

 

 

Google tells me this can be an issue with the pass4SymmKey, and to that end, I have updated the pass4SymmKey on both sides and restarted the instances a few times, to no avail. 

I'm stumped. Where did I go wrong that I can't get these search heads to cluster up nicely?

Labels (2)
0 Karma

loganac
Engager

I had this exact issue today and here's what I did:

For my issue, the SHC had a static captain. So I followed the Splunk docs to try and get them to become a RAFT distributed consensus voting for the captain. When I ran the commands the SHC cluster broke. After looking around for a while in the conf files I change two things on the non-captain servers.

In server.conf, the mgmt_uri was pointing to the existing captain. That has to be its own self per instructions in server.conf and delete the captain_url stanza. After I deleted those I restarted Splunk and ran the command pointed to the captain who was still the cluster

splunk add shcluster-member -current_member_uri <URI>:<management_port>

I repeated that for the other hosts until the captain was left

When I went to the captain I made sure that "mode = member" and deleted the captain_url stanza. When I restarted that host was no longer the captain and another had picked it up.

Hope this helps 

Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...