Monitoring Splunk

Unable to Control SPLUNKD

pfabrizi
Path Finder

I have been having space issue on one of my indexes running SPLUNK 6.5.1. The box appears to crash from time to time. Typically I have to restart SPLUNK, but today I am unable to control SPLUNKD. I get this when I issue a splunk command:

Stopping splunk helpers...
couldn't send SIGTERM to pid 2972: Operation not permitted
Couldn't send SIGTERM to some splunk helpers. [FAILED]
Error: Unable to stop splunk helpers.

What process\task is running this, I am guessing I can kill this?

Thanks!

Tags (1)
0 Karma

vinkumar_splunk
Splunk Employee
Splunk Employee

This could be because of stale PID files which reside under SPLUNK_HOME/var/run/splunk that causes splunk not to start.

Step 1:

rm $SPLUNK_HOME/var/run/splunk/splunkd.pid

Step 2:

$SPLUNK_HOME/bin/splunk status

Step 3:

$SPLUNK_HOME/bin/splunk start

cain1288
Explorer

This resolved my issue, thank you.

Not sure what the root cause was, and have not seen this before.

Steps to produce issue were:

1) yum update on RHEL 7 server

2) initiate reboot command (perhaps something here cause the issue? Splunkd did not completely stop?)

3) attempt to start splunk resulted in error: couldn't send sigterm to pid 332 operation not permitted

4) Resolution was to delete the /var/run/splunk/splunkd.pid as mentioned above, splunkd started on the next attempt

0 Karma

mcazacu
Engager

Fixed it for me. Thank you @vinkumar_splunk !!! 🙂

0 Karma

nickhills
Ultra Champion

Operation not permitted

suggests that you do not have permission to terminate the process.

Assuming your on linux, run:

ps -ef|grep splunkd

look to see which users are running the process named splunkd -p 8089 start (it will likely be root or splunk, and there may be two)
If you have one for each user, this is fine, but if they are both root, that will be your issue.

This can occur if someone forgets to run splunk as the splunk user (ie root restarts it) instead of sudoing to splunk.

If my comment helps, please give it a thumbs up!
0 Karma

pfabrizi
Path Finder

I am on Linux, I ran 'p's looking for splunkd and nothing running. I believe they did restart the server last night. I just checked and my root volume is out of space (at 99% used). Can that be the issue?

0 Karma

nickhills
Ultra Champion

Ahh, in that case it sound like Splunk is not running, but it was not stopped gracefully (perhaps because it ran out of disk) and thus it’s not cleared it’s pid.

I guess either that pid is no longer running, or (unluckily) something else now has it. Hence the ‘not permitted’

If my comment helps, please give it a thumbs up!
0 Karma

masonmorales
Influencer

Yes, you need to free up some disk space for Splunk to run properly.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...