How to stop/Kill Splunk forwarder after receiving error message

[ramprakash]

Hello Splunkers,

I am facing this issue since past one week.
Splunk is not forwarding any logs. I have tried everything to kill the Splunk process.
Below are the steps:

Receiving the error

Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
..........................................................................................................Timed out waiting for splunkd to stop.
The Splunk daemon (splunkd) is already running.

Tried to kill the process but no luck

[Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder/bin> ./splunk status
splunkd is running (PID: 9634304).
splunk helpers are running (PIDs: 18088142).

[Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder/bin> ps -ef | grep splunkd | grep -v grep
[Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder/bin>

[Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder/bin> ps -ef | grep -i splunk | grep -v grep
  splunk 18088142  9634304   0                  0:00 
  splunk  9175578 20316492   0 10:09:34  pts/1  0:00 -ksh
  splunk  8782682  9175578  14 17:40:01  pts/1  0:00 ps -ef

Removed the PID under var directory and started Splunk again.

[Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder/bin> ./splunk start

Splunk> Australian for grep.

Checking prerequisites...
        Checking mgmt port [8089]: already bound
ERROR: The mgmt port [8089] is already bound.  Splunk needs to use this port.
Would you like to change ports? [y/n]: n
Exiting....

I really appreciate if someone resolves this issue.

[jaime_ramirez]

Have you tried killing splunkd first and then its helpers, then make a clean start?:

 [Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder/bin> ./splunk status
 splunkd is running (PID: 9634304).
 splunk helpers are running (PIDs: 18088142).
[Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder/bin> echo "Killing splunkd" ; kill 9634304
[Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder/bin> echo "Killing helpers" ; kill 18088142
[Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder/bin> ./splunk start

Check the forwarder logs for any errors:

[Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder/bin> cd /APPLICATIONS/SPLUNK/currentversion/splunkforwarder/var/log/splunk
[Server@splunk]/APPLICATIONS/SPLUNK/currentversion/splunkforwarder//var/log/splunk> tail splunkd.log | egrep "ERROR|WARN"

Also check this link:
https://answers.splunk.com/answers/238155/splunk-process-compulsion-stop.html