topic Re: Search head cluster member does not come up after exec restart cmd from master in Deployment Architecture

Search head cluster member does not come up after exec restart cmd from master

immortalraghava — Wed, 08 Nov 2017 05:51:09 GMT

Hi,

I have a three search head SHC.

I see that one SHC member going for restart but never comes back up. This is the log line.

INFO SHCSlave - event=SHPSlave::handleHeartbeatDone master has instructed peer to restart

SHC has three members with Dynamic captain.

What could be going wrong.

Please help.

Re: Search head cluster member does not come up after exec restart cmd from master

mayurr98 — Wed, 08 Nov 2017 11:17:31 GMT

http://answers.splunk.com/answers/82275/why-is-my-windows-cluster-peer-node-continually-restarting

Essentially, directory permissions on /slave-apps/ on the search peer had been lost (why?) and directory was set to read only. As per the link above, resetting the permissions allowed the Cluster Master to once again populate the directory with the required apps.

Re: Search head cluster member does not come up after exec restart cmd from master

p_gurav — Wed, 08 Nov 2017 11:23:37 GMT

Hi immortalraghavan,
Can you please tell me output of "/splunk show shcluster-status" command?

Re: Search head cluster member does not come up after exec restart cmd from master

lycollicott — Thu, 09 Nov 2017 17:07:29 GMT

So many things could be going wrong...so very many.

What happens when you logon to that search head and just do splunk start?