Stopping splunk after upgrade to 6.3.3: "Could not kill pid"

[tobb]

Regarding Splunk Universal Forwarder 6.3.3
OS: AIX 7100-03-03-1415

Hello

I'm using a script to upgrade splunk forwarder on our servers.
The script is run from a single server and uses rsh to do the nessesary work.

In essence the script does the following:

#Disable boot-start
$output = `rsh $target /opt/splunkforwarder/bin/splunk disable boot-start 2>&1`;

#Stop Splunk
$output = `rsh $target /opt/splunkforwarder/bin/splunk stop 2>&1`;

#It then backs up some configuration files...

#Remove Splunk
$output = `rsh $target rm -r /opt/splunkforwarder 2>&1`;

#Extract Splunk
$output = `rsh $target "(cd /opt && tar -xf /tmp/$base)" 2>&1`;

#Re-deploy configuration files...

#Set permissions
$output = `rsh $target chown -R splunk:staff /opt/splunkforwarder/ 2>&1`;

#Start Splunk
$output = `rsh $target /opt/splunkforwarder/bin/splunk start --accept-license 2>&1`;

#Enable boot-start
$output = `rsh $target /opt/splunkforwarder/bin/splunk enable boot-start -user splunk 2>&1`;

The script runs without problems and we get the upgraded servers to start indexing. Everything looks to be working.

However, I'm having some trouble trying to stop splunk forwarder on the upgraded servers...

Basically when trying to run (as root):
$ /opt/splunkforwarder/bin/splunk stop
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
Could not kill pid 41156748.

A quick $ ps -ef | grep splunkd | grep -v grep shows:
root 41156748 splunkd -p 8089 start
root 41680960 [splunkd pid=41156748] splunkd -p 8089 start [process-runner]

And just to be sure $ whoami
root

Can someone please explain what's going on here, so we can fix this?

Thanks! 🙂

[emeelan_splunk]

I think you might find the answer to your problem here: https://answers.splunk.com/answers/40435/error-unable-to-stop-splunk-helpers.html

let me know if this helps!

[tobb]

I have tried that, but with the same result. I did it as root here just to illustrate that it's most likely not a permissions-problem.

test30:/opt/splunkforwarder/var/log/splunk> su splunk -c /opt/splunkforwarder/bin/splunk stop
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
Could not kill pid 13238426.

I would also add that the same thing happens when using the System Resource Controller directly
test30:/opt/splunkforwarder/var/log/splunk> stopsrc -s splunkd
0513-004 The Subsystem or Group, splunkd, is currently inoperative.

[skalliger]

Well, I would open a support case then to see if there are any known problems.

[skalliger]

Usually I would try to add root to the splunk group, in your case add root to staff. The question is, why are you stopping the process as root? I would atleast do sudo -u splunk -c 'the command needed'.

Skalli