Monitoring Splunk

Unable to Control SPLUNKD

pfabrizi
Path Finder

I have been having space issue on one of my indexes running SPLUNK 6.5.1. The box appears to crash from time to time. Typically I have to restart SPLUNK, but today I am unable to control SPLUNKD. I get this when I issue a splunk command:

Stopping splunk helpers...
couldn't send SIGTERM to pid 2972: Operation not permitted
Couldn't send SIGTERM to some splunk helpers. [FAILED]
Error: Unable to stop splunk helpers.

What process\task is running this, I am guessing I can kill this?

Thanks!

Tags (1)
0 Karma

vinkumar_splunk
Splunk Employee
Splunk Employee

This could be because of stale PID files which reside under SPLUNK_HOME/var/run/splunk that causes splunk not to start.

Step 1:

rm $SPLUNK_HOME/var/run/splunk/splunkd.pid

Step 2:

$SPLUNK_HOME/bin/splunk status

Step 3:

$SPLUNK_HOME/bin/splunk start

cain1288
Explorer

This resolved my issue, thank you.

Not sure what the root cause was, and have not seen this before.

Steps to produce issue were:

1) yum update on RHEL 7 server

2) initiate reboot command (perhaps something here cause the issue? Splunkd did not completely stop?)

3) attempt to start splunk resulted in error: couldn't send sigterm to pid 332 operation not permitted

4) Resolution was to delete the /var/run/splunk/splunkd.pid as mentioned above, splunkd started on the next attempt

0 Karma

mcazacu
Engager

Fixed it for me. Thank you @vinkumar_splunk !!! 🙂

0 Karma

nickhills
Ultra Champion

Operation not permitted

suggests that you do not have permission to terminate the process.

Assuming your on linux, run:

ps -ef|grep splunkd

look to see which users are running the process named splunkd -p 8089 start (it will likely be root or splunk, and there may be two)
If you have one for each user, this is fine, but if they are both root, that will be your issue.

This can occur if someone forgets to run splunk as the splunk user (ie root restarts it) instead of sudoing to splunk.

If my comment helps, please give it a thumbs up!
0 Karma

pfabrizi
Path Finder

I am on Linux, I ran 'p's looking for splunkd and nothing running. I believe they did restart the server last night. I just checked and my root volume is out of space (at 99% used). Can that be the issue?

0 Karma

nickhills
Ultra Champion

Ahh, in that case it sound like Splunk is not running, but it was not stopped gracefully (perhaps because it ran out of disk) and thus it’s not cleared it’s pid.

I guess either that pid is no longer running, or (unluckily) something else now has it. Hence the ‘not permitted’

If my comment helps, please give it a thumbs up!
0 Karma

masonmorales
Influencer

Yes, you need to free up some disk space for Splunk to run properly.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...