topic Re: Unable to Control SPLUNKD in Monitoring Splunk

Unable to Control SPLUNKD

pfabrizi — Tue, 05 Dec 2017 12:56:13 GMT

I have been having space issue on one of my indexes running SPLUNK 6.5.1. The box appears to crash from time to time. Typically I have to restart SPLUNK, but today I am unable to control SPLUNKD. I get this when I issue a splunk command:

Stopping splunk helpers...
couldn't send SIGTERM to pid 2972: Operation not permitted
Couldn't send SIGTERM to some splunk helpers. [FAILED]
Error: Unable to stop splunk helpers.

What process\task is running this, I am guessing I can kill this?

Thanks!

Re: Unable to Control SPLUNKD

nickhills — Tue, 05 Dec 2017 13:23:54 GMT

Operation not permitted

suggests that you do not have permission to terminate the process.

Assuming your on linux, run:

ps -ef|grep splunkd

look to see which users are running the process named splunkd -p 8089 start (it will likely be root or splunk, and there may be two)
If you have one for each user, this is fine, but if they are both root, that will be your issue.

This can occur if someone forgets to run splunk as the splunk user (ie root restarts it) instead of sudoing to splunk.

Re: Unable to Control SPLUNKD

pfabrizi — Tue, 05 Dec 2017 13:35:21 GMT

I am on Linux, I ran 'p's looking for splunkd and nothing running. I believe they did restart the server last night. I just checked and my root volume is out of space (at 99% used). Can that be the issue?

Re: Unable to Control SPLUNKD

masonmorales — Tue, 05 Dec 2017 18:21:50 GMT

Yes, you need to free up some disk space for Splunk to run properly.

Re: Unable to Control SPLUNKD

nickhills — Tue, 05 Dec 2017 18:27:40 GMT

Ahh, in that case it sound like Splunk is not running, but it was not stopped gracefully (perhaps because it ran out of disk) and thus it’s not cleared it’s pid.

I guess either that pid is no longer running, or (unluckily) something else now has it. Hence the ‘not permitted’

Re: Unable to Control SPLUNKD

vinkumar_splunk — Fri, 21 Sep 2018 07:25:52 GMT

This could be because of stale PID files which reside under SPLUNK_HOME/var/run/splunk that causes splunk not to start.

Step 1:

rm $SPLUNK_HOME/var/run/splunk/splunkd.pid

Step 2:

$SPLUNK_HOME/bin/splunk status

Step 3:

$SPLUNK_HOME/bin/splunk start

Re: Unable to Control SPLUNKD

mcazacu — Fri, 25 Dec 2020 02:37:10 GMT

Fixed it for me. Thank you @vinkumar_splunk !!! 🙂

Re: Unable to Control SPLUNKD

cain1288 — Thu, 28 Oct 2021 14:31:43 GMT

This resolved my issue, thank you.

Not sure what the root cause was, and have not seen this before.

Steps to produce issue were:

1) yum update on RHEL 7 server

2) initiate reboot command (perhaps something here cause the issue? Splunkd did not completely stop?)

3) attempt to start splunk resulted in error: couldn't send sigterm to pid 332 operation not permitted

4) Resolution was to delete the /var/run/splunk/splunkd.pid as mentioned above, splunkd started on the next attempt