JMS Modular Input app( http://splunk-base.splunk.com/apps/69562/jms-messaging-modular-input )
We are testing JMS Modular Input with Splunk on a Windows VM environment, and noticed that Splunk get error message when restarting Splunk due to splunkd management port was set is being used.
Monitoring Splunk processes on Task Manager while restaring Splunk, I've noticed java.exe which is used by the jms modular input does not stop. By killing the java.exe process, Splunk can start without the warning message.
I do not have this issue with Splunk running on a Solaris machine. What is going on?
The app author told us that, if the splunkd process has genuinely terminated, then this connection attempt loop should fail 3 times , then invoke System.exit() , the JVM suicide call.
Then, after analyzing this behavior more, I have determined part of the cause of this issue. After we issue the ./splunk stop command on Windows the java process takes a long time to close a connection and this seems to take a long time. The instance is running on Windows VM on Hyper V and we've noticed some other network connection issues that don't close in a timely manner. We're not a 100% sure about the root cause but the Hyper V network drivers might have something to do with this.
Either way, after about 5min we see that the process does kill itself. We don't see this issue on our production machine, which is Solaris. As such, we can close this issue as we're solving this issue here in dev by using "patience".