Deployment Architecture

Search Head Cluster - can't add members after captain bootstrap (8.1.2)?

whar_garbl
Path Finder

I am rebuilding a SH cluster from scratch. I've followed the documentation carefully to this point. I have the shcluster captain bootstrapped and splunk show shcluster-status shows the captain as the only member, but the bootstrapping process failed to add my member nodes due to comms errors. Pretty sure I've got those fixed now. 

When I do splunk add shcluster-member -current_member_uri https://ip-address-of-captain:8089 on a member node, it tells me: 

 

 

current_member_uri is pointing back to this same node. It should point to a node that is already a member of a cluster. 

 

 

Obviously, I have checked and re-checked the uri, which I believe is correct (https://ip-address-of-captain:8089), and that is set right in server.conf on both sides. There is no IP conflict and the servers have no issue communicating. 

If I run splunk add shcluster-member -new_member_uri https://ip-address-of-member:8089 from the captain, it tells me:

 

 

Failed to proxy call to member https://ip-address-of-member:8089

 

 

Google tells me this can be an issue with the pass4SymmKey, and to that end, I have updated the pass4SymmKey on both sides and restarted the instances a few times, to no avail. 

I'm stumped. Where did I go wrong that I can't get these search heads to cluster up nicely?

Labels (2)
0 Karma

loganac
Engager

I had this exact issue today and here's what I did:

For my issue, the SHC had a static captain. So I followed the Splunk docs to try and get them to become a RAFT distributed consensus voting for the captain. When I ran the commands the SHC cluster broke. After looking around for a while in the conf files I change two things on the non-captain servers.

In server.conf, the mgmt_uri was pointing to the existing captain. That has to be its own self per instructions in server.conf and delete the captain_url stanza. After I deleted those I restarted Splunk and ran the command pointed to the captain who was still the cluster

splunk add shcluster-member -current_member_uri <URI>:<management_port>

I repeated that for the other hosts until the captain was left

When I went to the captain I made sure that "mode = member" and deleted the captain_url stanza. When I restarted that host was no longer the captain and another had picked it up.

Hope this helps 

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...