Deployment Architecture

Unable to remove member from search head cluster

Lucas_K
Motivator

We need to migrate members in and out of a search head cluster.

It is documented here ( http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/Removeaclustermember ) that the command is "splunk remove shcluster-member" if performed on the host you want removed and "splunk remove shcluster-member -mgmt_uri :" if performed anywhere else.

When I try this all I get is

In handler 'shclustermemberconsensus': Failed to Set Configuration. One potential reason is captain could not hear back from all the nodes in a timeout period. Ensure all to be added nodes are up, and increase the raft timeout. If all nodes are up and running, look at splunkd.log for appendEntries errors due to mgmt_uri mismatch

Looking up that error points towards opening up the timings in server.conf

Setting them to these values below doesn't change the error.

[shclustering]
cxn_timeout_raft = 60
send_timeout_raft = 60
rcv_timeout_raft = 60
rep_cxn_timeout = 60
heartbeat_timeout = 120

Any ideas? splunkd.log gives no clues as to what to do.

ksahu
New Member

@Lucas_K Please share the solution if you were able to solve this? 

0 Karma

ThomasControlwa
Path Finder

hi, so I'm know is it is a late tra to answer, but ig ot the same issue.
in my case I fixed as following.

  • make sure that the member have connection to the deployer, and clustermaster as well.
  • List item
  • for me works : ::remove all "shcluster entries" at SPLUNK_HOME/etc/system/local/server.conf ::rm -rf /SPLUNK_HOME/etc/instance.cfg ::rm -rf /SPLUNK_HOME/var/run/splunk/_raft/* restart splunk, done now you can join or whatever you are looking for

additional links:
https://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/Handleraftissues
http://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/Removeaclustermember
http://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/SHCconfigurationoverview

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...