Deployment Architecture

Search head cluster member does not come up after exec restart cmd from master

Path Finder


I have a three search head SHC.

I see that one SHC member going for restart but never comes back up. This is the log line.

INFO SHCSlave - event=SHPSlave::handleHeartbeatDone master has instructed peer to restart

SHC has three members with Dynamic captain.

What could be going wrong.

Please help.

0 Karma


So many things could be going very many.

What happens when you logon to that search head and just do splunk start?

0 Karma


Hi immortalraghavan,
Can you please tell me output of "/splunk show shcluster-status" command?

0 Karma

Super Champion

Essentially, directory permissions on /slave-apps/ on the search peer had been lost (why?) and directory was set to read only. As per the link above, resetting the permissions allowed the Cluster Master to once again populate the directory with the required apps.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...