How to forcefully change the current search head captain to actual captain in the search head cluster Environment ?

[Hemnaath]

Hi All,

After deploying an customized app to all the clustered search head members via Deployer using the below command and the app got successfully deployed but I am getting the following message in the forwarder management console,

Command used to push the app from deployer.

/opt/splunk/bin/splunk apply shcluster-bundle --answer-yes -target https://searchhead1:8089 -auth admin:xxxxx

Message Detail:

Search peer searchhead1.xxx.com has the following message: Failed to synchronize configuration with KVStore cluster. replSetReconfig should only be run on PRIMARY, but my state is SECONDARY; use the "force" argument to override

By executing the show cluster status we could see that searchhead2 is assigned as captain.

/opt/splunk/bin/splunk show shcluster-status -auth admin:xxxx

/opt/splunk/bin/splunk transfer shcluster-captain -mgmt_uri https://searchhead1.xxx.com:8089 -auth admin:xxxxx

So kindly let me know whether, we need to execute the above command to forcefully to change current search head2 from captain to search head 1 as a captain.

[p_gurav · Accepted solution]

Hi,

You can run this command with SH2 on deployer :

/opt/splunk/bin/splunk apply shcluster-bundle --answer-yes -target https://searchhead2:8089 -auth admin:xxxxx

N even though you want to change the captaincy run the command:

/opt/splunk/bin/splunk transfer shcluster-captain -mgmt_uri https://searchhead1.xxx.com:8089 -auth admin:xxxxx

[Anam]

@hemnaath

If this answer was helpful and you were able to solve your question, please click accept on the answer.

Thanks

[duartet]

We had the cluster with the captain node stuck, the way we fixed it was by:

• stopping the splunk instance,
• waited for the new captain to be elected,
• start again the instance,

and when checking the shcluster-status, it was ok again.