Why does my Splunk instances go down and I am getting below error?

[uagraw01]

Hello Splunkers!!

One a everyday basis one of my Splunk instances goes down and i am getting below error. Please suggest me for the permanent fix and workaround for the below error.

splunkd 7081 was not running.
Stopping splunk helpers...

Done.
Stopped helpers.
Removing stale pid file... done.
splunkd is not running.

[tshah-splunk]

Hey @uagraw01,

Can you confirm if you have enabled boot-start to run splunk? If not, please consider setting it up using the below command. Ref doc - https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/ConfigureSplunktostartatboottime 

sudo $SPLUNK_HOME/bin/splunk enable boot-start -user splunk

 You can also configure your Splunk to run as systemd. Reference doc can be found here - https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/RunSplunkassystemdservice 

[uagraw01]

@tshah-splunk  Yes i already enabled the boot-start on the Splunk server with sudo user.

[isoutamo]

Actually I just realised that we have same issue on couple of IHF servers. Those all are used systemd to starting.  Need to check some parameters etc. first and see if those can lead this.

[isoutamo]

Hi

What you find from splunkd.log ($SPLUNK_HOME/var/log/splunk/)? There should be something which told the reason why it has stopped.

r. Ismo

[uagraw01]

 Below is the error i am getting whenever the instances will goes down.

10-21-2021 09:39:25.152 -0700 FATAL ProcessRunner - Unexpected EOF from process runner child!

10-21-2021 09:39:25.178 -0700 ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!

@isoutamo

[anem]

hi if u had found the solution pls post here

[uagraw01]

@anem In my case simply increased the cpu core of my HF. I have increased it from 8 core cpu to 16 core CPU. As of now my problem is resolved but may be in your it differs.