I have two splunk installation in this server. I see that both are running based on splunkd processed. When I try to stop one of the (forwarder) it fails to stop. No error messages anywhere that I can find. Only a warning about splunk_home being set, which seems unrelated.
any hints?
-fdo
[sfdc@adhoc-app1-11-sfm bin]$ pwd
/home/sfdc/apps/splunk/prod-forwarder/bin
[sfdc@adhoc-app1-11-sfm bin]$ ps -efH | grep -B3 -A3 splunkd
root 5987 1 0 Mar23 ? 00:00:00 /usr/bin/python -tt /usr/sbin/yum-updatesd
root 6001 1 0 Mar23 ? 00:00:05 /usr/libexec/gam_server
root 6495 1 0 Mar23 ? 00:00:00 udevd
sfdc 21156 1 11 Nov29 ? 13:56:52 splunkd -p 42200 start
sfdc 21157 21156 0 Nov29 ? 00:01:23 splunkd -p 42200 start
sfdc 8566 21157 0 19:06 ? 00:00:01 splunkd search --id=1354647977.1740 --maxbuckets=0 --ttl=60 --maxout=500000 --maxtime=0 --lookups=1 --reduce_freq=10 --user=perfeng --pro --roles=admin:can_delete:corda_user:custcorelog:large_storage:mandm_team:power:searchrelevancy:splunk_admin:splunk_corda_user:splunk_delete:splunk_large_storage:splunk_mandm_dev:splunk_power_user:user
sfdc 8567 8566 0 19:06 ? 00:00:00 splunkd search --id=1354647977.1740 --maxbuckets=0 --ttl=60 --maxout=500000 --maxtime=0 --lookups=1 --reduce_freq=10 --user=perfeng --pro --roles=admin:can_delete:corda_user:custcorelog:large_storage:mandm_team:power:searchrelevancy:splunk_admin:splunk_corda_user:splunk_delete:splunk_large_storage:splunk_mandm_dev:splunk_power_user:user
sfdc 21212 1 0 Nov29 ? 00:14:53 python -O /home/sfdc/apps/splunk/prod-datacenter-1-indexer/lib/python2.6/site-packages/splunk/appserver/mrsparkle/root.py start
sfdc 8494 1 5 Dec01 ? 04:25:50 splunkd -p 9779 start
sfdc 8496 8494 0 Dec01 ? 00:00:00 splunkd -p 9779 start
[sfdc@adhoc-app1-11-sfm bin]$ ./splunk stop
Warning: overriding $SPLUNK_HOME setting in environment ("/home/sfdc/apps/splunk/prod-datacenter-1-indexer") with "/home/sfdc/apps/splunk/prod-forwarder". If this is not correct, edit /home/sfdc/apps/splunk/prod-forwarder/etc/splunk-launch.conf
splunkweb is not running.
splunkd is not running. [FAILED]
There are two instances of splunk running.
The indexer is running on port 42200. Because it has child processes that are running searches, that instance is the indexer. It is running as process id 21156 and subprocesses.
The forwarder is running as process id 8494 on port 9779. It also has subprocesses.
You have an environment variable ($SPLUNK_HOME
) set. The message that you get is simply pointing out that Splunk is ignoring the environment variable (which is good, because $SPLUNK_HOME
is pointing to the indexer, not the forwarder.)
I would
su - sfdc
cd /home/sfdc/apps/splunk/prod-forwarder/bin
./splunk stop
And if that didn't work, I would ensure that all files belong to sfdc
chown -R sfdc /home/sfdc/apps/splunk/prod-forwarder
and try again. Depending on the what the forwarder is monitoring, there might not be any big consequences to just killing the forwarder processes, but I consider this a last resort.
HTH
It looks like you are trying to stop the wrong splunk instance. The warning given shows you are overriding the $SPLUNK_HOME
variable, which is not the forwarder location.
/home/sfdc/apps/splunk/prod-datacenter-1-indexer/bin/splunk stop