I have a Daily Splunk report that lets me know when it hasn't heard from a server for a while.
splunkd 48961 was not running.
Stopping splunk helpers...
[ OK ]
Done.
Stopped helpers.
Further checking shows that no crash file was generated. So what caused Splunk to crash?
For those interested, here is the query I use for the daily report:
| metadata type=hosts | eval age = now() - lastTime | search age > 86400 | sort age d | convert ctime(lastTime) | fields age,host,lastTime
If you want to use systemd refer to Splunk systemd unit file in versions 7.2.2 and newer - how do I stop this prompting for the root pas... the settings in the file there should ensure a clean startup/shutdown.
If not try init.d again...(as per woodcock's suggestion)
If you are running the hot (mess) new systemd
boot-start, the default does a kill -9
which causes all manner of terribleness including stale pid files. Switch back to init.d
for starters.
Check the splunkd log for any unusual behavior or any WARN/ERROR events.
Also it could be something regarding the ulimits. You could check the following info:
https://docs.splunk.com/Documentation/Splunk/7.3.0/Troubleshooting/ulimitErrors
here is what I found:
07-29-2019 12:17:42.721 -0500 FATAL ProcessRunner - Unexpected EOF from process runner child!
07-29-2019 12:17:42.721 -0500 ERROR ProcessRunner - helper process seems to have died (child killed by signal 15: Terminated)!
The next timestamp is from my restarting Splunk:
08-01-2019 13:12:58.573 -0500 INFO ServerConfig - My GUID is BD8........