topic Re: Why doe Splunk instances goes down and I am getting below error? in Getting Data In

Why does my Splunk instances go down and I am getting below error?

LogUx — Thu, 24 Feb 2022 14:23:56 GMT

Hello Splunkers!!

One a everyday basis one of my Splunk instances goes down and i am getting below error. Please suggest me for the permanent fix and workaround for the below error.

splunkd 7081 was not running.
Stopping splunk helpers...

Done.
Stopped helpers.
Removing stale pid file... done.
splunkd is not running.

Re: Admin

isoutamo — Thu, 28 Oct 2021 10:04:38 GMT

What you find from splunkd.log ($SPLUNK_HOME/var/log/splunk/)? There should be something which told the reason why it has stopped.

r. Ismo

Re: Admin

LogUx — Thu, 28 Oct 2021 15:12:53 GMT

Below is the error i am getting whenever the instances will goes down.

10-21-2021 09:39:25.152 -0700 FATAL ProcessRunner - Unexpected EOF from process runner child!

10-21-2021 09:39:25.178 -0700 ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!

@isoutamo

Re: Admin

anem — Thu, 24 Feb 2022 04:05:33 GMT

hi if u had found the solution pls post here

Re: Why doe Splunk instances goes down and I am getting below error?

tshah-splunk — Thu, 24 Feb 2022 06:21:20 GMT

Hey @LogUx,

Can you confirm if you have enabled boot-start to run splunk? If not, please consider setting it up using the below command. Ref doc - https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/ConfigureSplunktostartatboottime

sudo $SPLUNK_HOME/bin/splunk enable boot-start -user splunk

You can also configure your Splunk to run as systemd. Reference doc can be found here - https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/RunSplunkassystemdservice

Re: Why doe Splunk instances goes down and I am getting below error?

isoutamo — Thu, 24 Feb 2022 15:03:24 GMT

Actually I just realised that we have same issue on couple of IHF servers. Those all are used systemd to starting. Need to check some parameters etc. first and see if those can lead this.

Re: Admin

LogUx — Thu, 24 Feb 2022 15:23:18 GMT

@anem In my case simply increased the cpu core of my HF. I have increased it from 8 core cpu to 16 core CPU. As of now my problem is resolved but may be in your it differs.

Re: Why doe Splunk instances goes down and I am getting below error?

LogUx — Thu, 24 Feb 2022 15:25:46 GMT

@tshah-splunk Yes i already enabled the boot-start on the Splunk server with sudo user.