Why Splunkd stops running but the splunkd.pid file remains

[MikeBertelsen]

I have a Daily Splunk report that lets me know when it hasn't heard from a server for a while.

Sometimes when I get to the server I use my restartsplunk script and get one of these:

splunkd 48961 was not running.
Stopping splunk helpers...
[ OK ]
Done.
Stopped helpers.

Removing stale pid file... done.

Further checking shows that no crash file was generated. So what caused Splunk to crash?

For those interested, here is the query I use for the daily report:
| metadata type=hosts | eval age = now() - lastTime | search age > 86400 | sort age d | convert ctime(lastTime) | fields age,host,lastTime

[gjanders]

If you want to use systemd refer to Splunk systemd unit file in versions 7.2.2 and newer - how do I stop this prompting for the root pas... the settings in the file there should ensure a clean startup/shutdown.

If not try init.d again...(as per woodcock's suggestion)

[woodcock]

If you are running the hot (mess) new systemd boot-start, the default does a kill -9 which causes all manner of terribleness including stale pid files. Switch back to init.d for starters.

[jaime_ramirez]

Check the splunkd log for any unusual behavior or any WARN/ERROR events.

Also it could be something regarding the ulimits. You could check the following info:
https://docs.splunk.com/Documentation/Splunk/7.3.0/Troubleshooting/ulimitErrors

[MikeBertelsen]

here is what I found:
07-29-2019 12:17:42.721 -0500 FATAL ProcessRunner - Unexpected EOF from process runner child!
07-29-2019 12:17:42.721 -0500 ERROR ProcessRunner - helper process seems to have died (child killed by signal 15: Terminated)!

The next timestamp is from my restarting Splunk:
08-01-2019 13:12:58.573 -0500 INFO ServerConfig - My GUID is BD8........

[jaime_ramirez]

Check this:
https://answers.splunk.com/answers/476273/why-do-forwarders-go-down-with-unexpected-eof-mess.html