Unable to bind Splunk4Jmx with WAS

[user_malathi]

Hi ,
I have installed Splunk4JMX and modified the config file with required host,jmxport,jmxserviceurl,jmxuser,jmxpass. And Also updated the JAVA_HOME and SPLUNK_HOME path

<jmxserver jmxServiceURL="service:jmx:iiop://xxxxx:xxxx/jndi/JMXConnector" host="xxxx" jmxport="xxx" jvmDescription="xxx" jmxuser="xxxx" jmxpass="xxx">

Also I am using JDK 1.8 version and WAS 7 version.
To work with WAS , I have added the below Jar files in the /SPLUNK4JMX/bin/lib directory
com.ibm.ws.orb_8.5.0.jar,com.ibm.ws.ejb.thinclient_8.5.0.jar,com.ibm.ws.admin.client_8.5.0.jar
ibmorbapi.jar,ibmorb.jar,ibmcfw.jar in /SPLUNK4JMX/bin/lib/boot directory

I have tried with RMI/SOAP port.But when i run the command,
$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config jmx | $SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/bin/python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py

I got teh below error,

[Fatal Error] :-1:-1: Premature end of file.
ERROR Can't connect to Splunk REST API with the token [Splunk uiZQJ1gcYYetyDXz6bm6ED^TA2EdiCplLKjKICU^dGGAMCOuzYIRFhXv0dPlv4MWH_IE4276UkjnoCHRreWYTNGhTWLeJmTGXsXc1Woe_XA8vkHmwItpraCu2ko], either the token is invalid or SplunkD has exited : HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="WARN">call not properly authenticated</msg>
</messages>
</response>

JSAS1480I: Security is not enabled because the ConfigURL property file is not set.
Exception in thread "Thread-4" java.lang.NoClassDefFoundError: sun/io/ByteToCharConverter
at com.ibm.rmi.iiop.TcsToCharConverter$1.run(TcsToCharConverter.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.rmi.iiop.TcsToCharConverter.getSunConverter(TcsToCharConverter.java:144)
at com.ibm.rmi.iiop.TcsToCharConverter.convertFromByte(TcsToCharConverter.java:95)
at com.ibm.rmi.iiop.CDRReader.readStringOrIndirection(CDRReader.java:536)
at com.ibm.rmi.iiop.CDRReader.read_string(CDRReader.java:521)
at com.ibm.rmi.IOR.read(IOR.java:361)
at com.ibm.rmi.iiop.Connection._locate(Connection.java:546)
at com.ibm.rmi.iiop.Connection.locate(Connection.java:505)
at com.ibm.rmi.iiop.GIOPImpl.locate(GIOPImpl.java:240)
at com.ibm.rmi.corba.Corbaloc.locateUsingINS(Corbaloc.java:307)
at com.ibm.rmi.corba.Corbaloc.resolve(Corbaloc.java:378)
at com.ibm.rmi.corba.ORB.objectURLToObject(ORB.java:3805)
at com.ibm.CORBA.iiop.ORB.objectURLToObject(ORB.java:3377)
at com.ibm.rmi.corba.ORB.string_to_object(ORB.java:3700)
at com.ibm.ws.naming.util.WsnInitCtxFactory.stringToObject(WsnInitCtxFactory.java:1645)
at com.ibm.ws.naming.util.WsnInitCtxFactory.getCosRootContext(WsnInitCtxFactory.java:1338)
at com.ibm.ws.naming.util.WsnInitCtxFactory.getRootContextFromServer(WsnInitCtxFactory.java:1088)
at com.ibm.ws.naming.util.WsnInitCtxFactory.getRootJndiContext(WsnInitCtxFactory.java:939)
at com.ibm.ws.naming.util.WsnInitCtxFactory.getInitialContextInternal(WsnInitCtxFactory.java:614)
at com.ibm.ws.naming.util.WsnInitCtx.getContext(WsnInitCtx.java:128)
at com.ibm.ws.naming.util.WsnInitCtx.getContextIfNull(WsnInitCtx.java:765)
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:164)
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:179)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at com.ibm.websphere.management.remote.WsProvider.newJMXConnector(WsProvider.java:139)
at javax.management.remote.JMXConnectorFactory.getConnectorAsService(JMXConnectorFactory.java:476)
at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:344)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:269)
at com.splunk.modinput.jmx.JMXModularInput$ProcessServerThread.connect(Unknown Source)
at com.splunk.modinput.jmx.JMXModularInput$ProcessServerThread.run(Unknown Source)
ERROR It has been determined via the REST API that all inputs have been disabled.

I am not sure what is missed here as i am new to the SPLUNK.Please help me to resolve the issue

Thanks in advance.

[user_malathi]

Hi Damien,
I have verified with the given config.xsd and found that my config.xml is valid .
I noticed in the JMX Connectivity Status menu as Error and it didn't show the host name which is mentioned in the config.xml.
Inspect of this, it shows the local host name. Except hostname, it displays the correct port number and description mentioned in the config.xml.
I hope it took the host name from somewhere or it will be overwritten.Please help me to get the result without error

Thanks

[user_malathi]

Hi Damien,
Now i got the following error by execting command
index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" ExecProcessor error jmx.py

05-19-2015 08:26:50.492 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py" The root config object(JMXPoller) failed to initialize, stanza : jmx://helloworld

[Damien_Dallimor]

Maybe you have malformed your config.xml file.

I would suggest validating your XML against the XSD (config.xsd).

https://github.com/damiendallimore/SplunkModularInputsJavaFramework/blob/master/jmx/schema/config.xs...

[user_malathi]

Many Thanks for your reply Damien.

I looked into the WAS JAR and JRE Version Compatability table and confirmed that Oracle JRE 7 is compatible with WAS Jar 8. I did the same .Now i got a different error

$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config jmx | $SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/bin/python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py
[Fatal Error] :-1:-1: Premature end of file.
ERROR Can't connect to Splunk REST API with the token [Splunk xYrXXOHiiyPIXq6XYwBTRe^KxgA1kfcBJ2DLZpbKVsptuATZciaTGpL5StuLigItvBo5PPy0dFrlxHA7Sxkgl6FCH8TIU1MYzMqjM9Wi2hSdJUQHWm0L], either the token is invalid or SplunkD has exited : HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="WARN">call not properly authenticated</msg>
</messages>
</response>

JSAS1480I: Security is not enabled because the ConfigURL property file is not set.
ERROR It has been determined via the REST API that all inputs have been disabled.

Also i don't find any details related to jmx.py in the /var/log/splunk/splunkd.log . Is it mean that JMX.PY not running after the JRE version is changed?
Also please let me know why the error "ERROR It has been determined via the REST API that all inputs have been disabled" shown and how to resolve it.

Thanks for your help!

[Damien_Dallimor]

jmx.py is not designed to be run alone like you are attempting to do.
It is a Modular Input that must be run under the context of Splunk.
Hence the error mesages above.

[user_malathi]

Ok Damien.I got new error after the versionis changed.The error is

JSAS1480I: Security is not enabled because the ConfigURL property file is not set.
I am clueless about what i did wrong. Please let me know if you have any idea to resolve it.

Thanks!

[Damien_Dallimor]

I have no idea if that message is an ERROR , WARN , INFO, DEBUG etc...

That message is not from the JMX Mod Input.

Have you tried Googling "JSAS1480I: Security is not enabled because the ConfigURL property file is not set"

[user_malathi]

Hi Damien,
I referred the link ;http://www.errbay.com/?r=IBM-WAS-08.5&c=JSAS1480I and confirmed that the above text is an information message and not an error.Also it describes The com.ibm.CORBA.ConfigURL property specified in setupCmdLine.bat or on the java command line is not set.

Do we need to concentrate on this as it is not an error message. I know the only option to check teh jmx setup is successful or not by running the command $SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config jmx | $SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/bin/python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py.

Is there any option to check the jmx connectivity successful or not.
Also please let me know how to proceed further

Thanks!

[Damien_Dallimor]

Any errors will be searchable in Splunk

index=_internal ExecProcessor error jmx.py

[user_malathi]

Unfortunately there is no details updated when i run the above search in splunk. Also splunkd.log is not updated with the jmx related details.
Is it mean that the JMX not running Damien ?
Please advise !

[Damien_Dallimor]

Have you enabled your JMX stanza in Splunk ? (Browse to Data Inputs -> JMX)

[user_malathi]

Yes Damien. I have enabled the JMX Stanza. Still I have no error in splunkd.log related to jmx.py .
I got error only when i run the command

[Fatal Error] :-1:-1: Premature end of file.
ERROR Can't connect to Splunk REST API with the token [Splunk TEQ0iCckSwkEDLh168OAzBaM5EVvGuG3LF7A1RZK4Qyv73tGocwi2YBQdFR9SWzC0NKDEMDzKuapFOq9J951s3z9kdeVIBgn1XD1lwz9tv9d5BcfZ5vkfrTpz50], either the token is invalid or SplunkD has exited : HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="WARN">call not properly authenticated</msg>
</messages>
</response>

JSAS1480I: Security is not enabled because the ConfigURL property file is not set.
ERROR It has been determined via the REST API that all inputs have been disabled.

I triend the below search....
index=_internal ExecProcessor error jmx.py.
But it returns no error related to jmx.py in splunkd.log

[user_malathi]

Hi Damien,
I checked that there is no errors found in the splunk search when i run the command index=_internal ExecProcessor error jmx.py and also i couldn't see any output in the default JVM Metrics.Could you please try to find the root cause or solution to it.

[Damien_Dallimor · Accepted solution]

ERROR Can't connect to Splunk REST API with the token

This is a valid message if you are attempting to run the jmx.py script directly

Also I am using JDK 1.8 version and WAS 7 version

java.lang.NoClassDefFoundError: sun/io/ByteToCharConverter

I'm going to guess here that the WAS client jars are possibly not compatible with Oracle Java 8. Google "sun.io.bytetocharconverter" , this class is deprecated.
Perhaps you need to use a different JVM for those Jars you are loading , maybe an IBM JVM ? I'm sure the WAS IBM docs have this compatibility information.

[user_malathi]

Many Thanks for your reply Damien.

I looked into the WAS JAR and JRE Version Compatability table and confirmed that Oracle JRE 7 is compatible with WAS Jar 8. I did the same .Now i got a different error

$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config jmx | $SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/bin/python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py
[Fatal Error] :-1:-1: Premature end of file.
ERROR Can't connect to Splunk REST API with the token [Splunk xYrXXOHiiyPIXq6XYwBTRe^KxgA1kfcBJ2DLZpbKVsptuATZciaTGpL5StuLigItvBo5PPy0dFrlxHA7Sxkgl6FCH8TIU1MYzMqjM9Wi2hSdJUQHWm0L], either the token is invalid or SplunkD has exited : HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="WARN">call not properly authenticated</msg>
</messages>
</response>

JSAS1480I: Security is not enabled because the ConfigURL property file is not set.
ERROR It has been determined via the REST API that all inputs have been disabled.

Also i don't find any details related to jmx.py in the /var/log/splunk/splunkd.log . Is it mean that JMX.PY not running after the JRE version is changed?
Also please let me know why the error "ERROR It has been determined via the REST API that all inputs have been disabled" shown and how to resolve it.

Thanks for your help!