Monitoring Splunk

Unable to Control SPLUNKD

pfabrizi
Path Finder

I have been having space issue on one of my indexes running SPLUNK 6.5.1. The box appears to crash from time to time. Typically I have to restart SPLUNK, but today I am unable to control SPLUNKD. I get this when I issue a splunk command:

Stopping splunk helpers...
couldn't send SIGTERM to pid 2972: Operation not permitted
Couldn't send SIGTERM to some splunk helpers. [FAILED]
Error: Unable to stop splunk helpers.

What process\task is running this, I am guessing I can kill this?

Thanks!

Tags (1)
0 Karma

vinkumar_splunk
Splunk Employee
Splunk Employee

This could be because of stale PID files which reside under SPLUNK_HOME/var/run/splunk that causes splunk not to start.

Step 1:

rm $SPLUNK_HOME/var/run/splunk/splunkd.pid

Step 2:

$SPLUNK_HOME/bin/splunk status

Step 3:

$SPLUNK_HOME/bin/splunk start

cain1288
Explorer

This resolved my issue, thank you.

Not sure what the root cause was, and have not seen this before.

Steps to produce issue were:

1) yum update on RHEL 7 server

2) initiate reboot command (perhaps something here cause the issue? Splunkd did not completely stop?)

3) attempt to start splunk resulted in error: couldn't send sigterm to pid 332 operation not permitted

4) Resolution was to delete the /var/run/splunk/splunkd.pid as mentioned above, splunkd started on the next attempt

0 Karma

mcazacu
Engager

Fixed it for me. Thank you @vinkumar_splunk !!! 🙂

0 Karma

nickhills
Ultra Champion

Operation not permitted

suggests that you do not have permission to terminate the process.

Assuming your on linux, run:

ps -ef|grep splunkd

look to see which users are running the process named splunkd -p 8089 start (it will likely be root or splunk, and there may be two)
If you have one for each user, this is fine, but if they are both root, that will be your issue.

This can occur if someone forgets to run splunk as the splunk user (ie root restarts it) instead of sudoing to splunk.

If my comment helps, please give it a thumbs up!
0 Karma

pfabrizi
Path Finder

I am on Linux, I ran 'p's looking for splunkd and nothing running. I believe they did restart the server last night. I just checked and my root volume is out of space (at 99% used). Can that be the issue?

0 Karma

nickhills
Ultra Champion

Ahh, in that case it sound like Splunk is not running, but it was not stopped gracefully (perhaps because it ran out of disk) and thus it’s not cleared it’s pid.

I guess either that pid is no longer running, or (unluckily) something else now has it. Hence the ‘not permitted’

If my comment helps, please give it a thumbs up!
0 Karma

masonmorales
Influencer

Yes, you need to free up some disk space for Splunk to run properly.

0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...