Re-add search head.

[siemsplunk]

I tried to run ./splunk remove shcluster-member -mgmt_uri https://<CAPTAIN_IP>:8089 on the non-captain search head, which was successful. 

But on the re-election of the new captain with this command, it gave me an error. I run the command. 

./splunk add shcluster-member -mgmt_uri https://<NEW_CAPTAIN>:8089 -current_member_uri https://<PREV_CAPTAIN>:8089


WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Argument "mgmt_uri" is not supported by this handler.



But now, when I run the command ./splunk show shcluster-status --verbose on the new captain, I see the previous captain is no longer in the member section. 


If anyone could help, I would appreciate it. 

[isoutamo]

Hi

based on your steps you have probably do something else that you have planned?

Your steps said that you have removed your Captain on SHC and then you try to add another member to you SHC with using your previous Captain, which aren't anymore member of that SHC. It's obviously that it cannot work that way.

After your first step your SHC has elected a new Captain. You can see it from MC or any other member of SHC. Use GUI or cli to see that like 

splunk show shcluster-status --verbose

This told to you which is a new captain of your SHC.

If you are trying to move captain to another member then you should use 

splunk transfer shcluster-captain -mgmt_uri <URI>:<management_port> 

run this on current captain and use a new captain as mgmt_uri parameter.

r. Ismo 

[livehybrid]

Hi @siemsplunk 

If you are running this on the previous captain? If so you do not need to specify the mgmt_uri argument.

Check out https://docs.splunk.com/Documentation/Splunk/9.4.0/DistSearch/Addaclustermember#:~:text=rejoining%20... which covers the commands to add a SH into the cluster.

In short:

When running the splunk add command on the new member itself, use this version of the command:

splunk add shcluster-member -current_member_uri <URI>:<management_port>
Note the following:

current_member_uri is the management URI and port of any current member of the cluster that this node is joining. This parameter allows the new node to communicate with the cluster.


When running the splunk add command from a current cluster member, use this version of the command:

splunk add shcluster-member -new_member_uri <URI>:<management_port>

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[siemsplunk]

When I run ./splunk add shcluster-member -new_member_uri https://<CAPTAIN_IP>:8089

I get

Failed to proxy call to member https://<CAPTAIN_IP>:8089. ERROR: Node splsearch02 is already part of cluster id=2A5DDFE0-B873-4201-8B68-D2ACB4873DA7. A node cannot be part of two clusters. If you want to re-purpose this node, run 'splunk clean all' to clean this instance and then add to the cluster.

[livehybrid]

Hi @siemsplunk 

Could you try that command again but use current_member_uri instead?

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[siemsplunk]

o, sorry for the late reply.

If I run that one -current_member_uri     I get below issue

Node splsearch01 is already part of cluster id=2A5DDFE0-B873-4201-8B68-D2ACB4873DA7. To add a new member via this node use new_member_uri. Run 'splunk help add shcluster-member' for more info.

[siemsplunk · Accepted solution]

Thank you so much for the assistance, Below are the steps I used

I checked on sh02  /opt/splunk/etc/system/default and I observed preferred_captain = true Since I wanted to make this instance as a captain, I left that as it was. Ran the same on sh01 and sh03 and got the same result. Therefore, I navigated to /opt/splunk/etc/system/local/server.conf and under the [shclustering] stanza added the preferred_captain = false on both instances.

-> I run this command on the SH02 to make this as a static captain - "splunk edit shcluster-config -mode captain -captain_uri <URI>:<management_port> -election false"

-> I run this command on sh01 and sh03 - "splunk edit shcluster-config -mode member -captain_uri <URI>:<management_port> -election false"