One a everyday basis one of my Splunk instances goes down and i am getting below error. Please suggest me for the permanent fix and workaround for the below error.
splunkd 7081 was not running.
Stopping splunk helpers...
Removing stale pid file... done.
splunkd is not running.
Can you confirm if you have enabled boot-start to run splunk? If not, please consider setting it up using the below command. Ref doc - https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/ConfigureSplunktostartatboottime
sudo $SPLUNK_HOME/bin/splunk enable boot-start -user splunk
You can also configure your Splunk to run as systemd. Reference doc can be found here - https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/RunSplunkassystemdservice
Actually I just realised that we have same issue on couple of IHF servers. Those all are used systemd to starting. Need to check some parameters etc. first and see if those can lead this.
Below is the error i am getting whenever the instances will goes down.
10-21-2021 09:39:25.152 -0700 FATAL ProcessRunner - Unexpected EOF from process runner child!
10-21-2021 09:39:25.178 -0700 ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!