Deployment Architecture

Search Head Cluster - can't add members after captain bootstrap (8.1.2)?

whar_garbl
Path Finder

I am rebuilding a SH cluster from scratch. I've followed the documentation carefully to this point. I have the shcluster captain bootstrapped and splunk show shcluster-status shows the captain as the only member, but the bootstrapping process failed to add my member nodes due to comms errors. Pretty sure I've got those fixed now. 

When I do splunk add shcluster-member -current_member_uri https://ip-address-of-captain:8089 on a member node, it tells me: 

 

 

current_member_uri is pointing back to this same node. It should point to a node that is already a member of a cluster. 

 

 

Obviously, I have checked and re-checked the uri, which I believe is correct (https://ip-address-of-captain:8089), and that is set right in server.conf on both sides. There is no IP conflict and the servers have no issue communicating. 

If I run splunk add shcluster-member -new_member_uri https://ip-address-of-member:8089 from the captain, it tells me:

 

 

Failed to proxy call to member https://ip-address-of-member:8089

 

 

Google tells me this can be an issue with the pass4SymmKey, and to that end, I have updated the pass4SymmKey on both sides and restarted the instances a few times, to no avail. 

I'm stumped. Where did I go wrong that I can't get these search heads to cluster up nicely?

Labels (2)
0 Karma

loganac
Engager

I had this exact issue today and here's what I did:

For my issue, the SHC had a static captain. So I followed the Splunk docs to try and get them to become a RAFT distributed consensus voting for the captain. When I ran the commands the SHC cluster broke. After looking around for a while in the conf files I change two things on the non-captain servers.

In server.conf, the mgmt_uri was pointing to the existing captain. That has to be its own self per instructions in server.conf and delete the captain_url stanza. After I deleted those I restarted Splunk and ran the command pointed to the captain who was still the cluster

splunk add shcluster-member -current_member_uri <URI>:<management_port>

I repeated that for the other hosts until the captain was left

When I went to the captain I made sure that "mode = member" and deleted the captain_url stanza. When I restarted that host was no longer the captain and another had picked it up.

Hope this helps 

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...