I have SplunkLightForwarder turned on for AIX and Linux (Suse) and when initiating a Splunk stop it takes 5-10 minutes. The only reason I need to stop/start is because of the (3) splunkd daemons running (one too many) which in turn doesn't allow a newly installed Deployclient to query/copy/install the new .conf files pulled from the Deployment server. (is this a bug?)
So....Big picture....why does "Splunk stop" take so long?
I sometimes have the same problem as you, and we're also running AIX. I'm curious to see what responses you get to this post.
I don't know if this is related or not (probably not), but I noticed the 5-10 minute stop times on Splunk didn't happen as often after we moved to AIX 6.1. That's probably a coincidence, but I thought I'd point it out.
I just experienced this and recently sorted out my issue with support.
Our issue stemmed from what license file was installed on the forwarder. As you stated above, when upgrading an old forwarder, shutdown times would take forever and there were 3 daemons started. After some exploration, we found the enterprise trial license was set as the current license on each machine. Each splunk license is also shipped with a forwarding license, simply replacing the splunk.license file under $SPLUNK_HOME/etc/ with the splunk-forwarder.license resolved the shutdown issue and the extra daemon.
To give it a shot, simply log on to the forwarder, cd $SPLUNK_HOME/etc cp splunk-forwarder.license splunk.license
Let me know if this worked for you as well!
After further investigation, I can still recreate the 3 daemon pids by pushing the SplunkLightForwarder app to a remote machine and have the forwarder set to restart. Will update this thread if we find a solution.
After trouble shooting with support, we did come across a known bug in 4.1.4 with the restart hanging up the forwarders. It's slated to be fixed in 4.1.5