We need to migrate members in and out of a search head cluster.
It is documented here ( http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/Removeaclustermember ) that the command is "splunk remove shcluster-member" if performed on the host you want removed and "splunk remove shcluster-member -mgmt_uri :" if performed anywhere else.
When I try this all I get is
In handler 'shclustermemberconsensus': Failed to Set Configuration. One potential reason is captain could not hear back from all the nodes in a timeout period. Ensure all to be added nodes are up, and increase the raft timeout. If all nodes are up and running, look at splunkd.log for appendEntries errors due to mgmt_uri mismatch
Looking up that error points towards opening up the timings in server.conf
Setting them to these values below doesn't change the error.
cxn_timeout_raft = 60
send_timeout_raft = 60
rcv_timeout_raft = 60
rep_cxn_timeout = 60
heartbeat_timeout = 120
Any ideas? splunkd.log gives no clues as to what to do.
hi, so I'm know is it is a late tra to answer, but ig ot the same issue.
in my case I fixed as following.