Why did the JMS Messaging Modular Input disappear from the Inputs list with error "Can't connect to Splunk REST API with the token..."

[ehudb]

Hello

We have an issue with the JMS Messaging Modular Input 1.5, (Splunk 6.4)

The input worked well for a few days, but suddenly disappeared from "Inputs" list.
Now it's still working well behind the scenes, but we can't see it in the "Inputs" list.

This is the 'jms.py' part from splunkd.log (I have replaced the real name with 'splunkserver.domain'):

05-22-2016 10:31:25.579 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py" Can't connect to Splunk REST API with the token [Splunk 7edPTpv683LdKyzHTdJdpKe7^3M8HNZOUqOSCcsoq2ENb4m2Wkws_MM^JiopdX__HLusYRfAJueILXCCZvSbvTyfowOP7DIj^rl6n6kVsTgtO5ploGzAuC], either the token is invalid or SplunkD has exited : Connection reset
05-22-2016 10:31:29.459 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py" Probing socket connection to SplunkD failed.Either SplunkD has exited ,or if not,  check that your DNS configuration is resolving your system's hostname (splunkserver.domain) correctly : Connection refused
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py" Stanza jms://queue/:teva.ETL.TEST.msgTrace : Error connecting : javax.naming.NoInitialContextException: Cannot instantiate class: com.sun.jndi.fscontext.RefFSContextFactory [Root exception is java.lang.ClassNotFoundException: com.sun.jndi.fscontext.RefFSContextFactory]
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:674)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at javax.naming.InitialContext.init(InitialContext.java:242)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at javax.naming.InitialContext.<init>(InitialContext.java:216)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at com.splunk.modinput.jms.JMSModularInput$MessageReceiver.connect(Unknown Source)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at com.splunk.modinput.jms.JMSModularInput$MessageReceiver.run(Unknown Source)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py" Caused by: java.lang.ClassNotFoundException: com.sun.jndi.fscontext.RefFSContextFactory
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at java.security.AccessController.doPrivileged(Native Method)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at java.lang.Class.forName0(Native Method)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at java.lang.Class.forName(Class.java:266)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:63)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:671)
05-22-2016 10:31:29.888 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py"        ... 5 more
05-22-2016 10:31:40.147 +0300 INFO  ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/jms_ta/bin/jms.py
05-22-2016 10:31:55.848 +0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jms_ta/bin/jms.py" It has been determined via the REST API that all inputs have been disabled

I believe the problem comes from :

Can't connect to Splunk REST API with the token [Splunk 7edPTpv683LdKyzHTdJdpKe7^3M8HNZOUqOSCcsoq2ENb4m2Wkws_MM^JiopdX__HLusYRfAJueILXCCZvSbvTyfowOP7DIj^rl6n6kVsTgtO5ploGzAuC], either the token is invalid or SplunkD has exited : Connection reset

We have valid connection to 'splunkserver.local:8089', but it still seems to be an issue for it.
Is there any way to reset the modular input state?

[ehudb · Accepted solution]

Fixed by uninstalling "DB Connect 1" , apparently it does not go well with Splunk 6.4, and it's stopping other modular inputs from running.
DB Connect 2 might be the fix to this.

[ehudb]

Ok, it seems those messages came while I was restarting the Splunk service.
So if we stop the splunk service for few minutes, clean the logs, and
Then, starting the splunk we still have the issue but we don't see any jms.py errors in the logs.

[jkat54]

yeah the error will occur during restart because the token will change after restarting.

[jkat54]

Can you post your inputs.conf from within the jms app?