topic splunk stop fails in Monitoring Splunk

splunk stop fails

fcastano — Tue, 04 Dec 2012 20:27:34 GMT

I have two splunk installation in this server. I see that both are running based on splunkd processed. When I try to stop one of the (forwarder) it fails to stop. No error messages anywhere that I can find. Only a warning about splunk_home being set, which seems unrelated.

any hints?

-fdo

[sfdc@adhoc-app1-11-sfm bin]$ pwd
/home/sfdc/apps/splunk/prod-forwarder/bin
[sfdc@adhoc-app1-11-sfm bin]$ ps -efH | grep -B3 -A3 splunkd
root      5987     1  0 Mar23 ?        00:00:00   /usr/bin/python -tt /usr/sbin/yum-updatesd
root      6001     1  0 Mar23 ?        00:00:05   /usr/libexec/gam_server
root      6495     1  0 Mar23 ?        00:00:00   udevd
sfdc     21156     1 11 Nov29 ?        13:56:52   splunkd -p 42200 start
sfdc     21157 21156  0 Nov29 ?        00:01:23     splunkd -p 42200 start
sfdc      8566 21157  0 19:06 ?        00:00:01       splunkd search --id=1354647977.1740 --maxbuckets=0 --ttl=60 --maxout=500000 --maxtime=0 --lookups=1 --reduce_freq=10 --user=perfeng --pro --roles=admin:can_delete:corda_user:custcorelog:large_storage:mandm_team:power:searchrelevancy:splunk_admin:splunk_corda_user:splunk_delete:splunk_large_storage:splunk_mandm_dev:splunk_power_user:user
sfdc      8567  8566  0 19:06 ?        00:00:00         splunkd search --id=1354647977.1740 --maxbuckets=0 --ttl=60 --maxout=500000 --maxtime=0 --lookups=1 --reduce_freq=10 --user=perfeng --pro --roles=admin:can_delete:corda_user:custcorelog:large_storage:mandm_team:power:searchrelevancy:splunk_admin:splunk_corda_user:splunk_delete:splunk_large_storage:splunk_mandm_dev:splunk_power_user:user
sfdc     21212     1  0 Nov29 ?        00:14:53   python -O /home/sfdc/apps/splunk/prod-datacenter-1-indexer/lib/python2.6/site-packages/splunk/appserver/mrsparkle/root.py start
sfdc      8494     1  5 Dec01 ?        04:25:50   splunkd -p 9779 start
sfdc      8496  8494  0 Dec01 ?        00:00:00     splunkd -p 9779 start
[sfdc@adhoc-app1-11-sfm bin]$ ./splunk stop
Warning: overriding $SPLUNK_HOME setting in environment ("/home/sfdc/apps/splunk/prod-datacenter-1-indexer") with "/home/sfdc/apps/splunk/prod-forwarder".  If this is not correct, edit /home/sfdc/apps/splunk/prod-forwarder/etc/splunk-launch.conf
splunkweb is not running.
splunkd is not running.                                    [FAILED]

Re: splunk stop fails

alacercogitatus — Tue, 04 Dec 2012 20:45:31 GMT

It looks like you are trying to stop the wrong splunk instance. The warning given shows you are overriding the $SPLUNK_HOME variable, which is not the forwarder location.

/home/sfdc/apps/splunk/prod-datacenter-1-indexer/bin/splunk stop

Re: splunk stop fails

lguinn2 — Tue, 04 Dec 2012 21:44:05 GMT

There are two instances of splunk running.

The indexer is running on port 42200. Because it has child processes that are running searches, that instance is the indexer. It is running as process id 21156 and subprocesses.

The forwarder is running as process id 8494 on port 9779. It also has subprocesses.

You have an environment variable ($SPLUNK_HOME) set. The message that you get is simply pointing out that Splunk is ignoring the environment variable (which is good, because $SPLUNK_HOME is pointing to the indexer, not the forwarder.)

I would

su - sfdc
cd /home/sfdc/apps/splunk/prod-forwarder/bin
./splunk stop

And if that didn't work, I would ensure that all files belong to sfdc

chown -R sfdc /home/sfdc/apps/splunk/prod-forwarder

and try again. Depending on the what the forwarder is monitoring, there might not be any big consequences to just killing the forwarder processes, but I consider this a last resort.

HTH