Splunk 4 JMX: binding to a WAS 6.1

[stefano_guidoba]

Hi,
I installed Splunk for JMX in order to monitor JVM metrics on my WAS servers.
I configured the application to bind to a QA WAS.
I set the host, port, user and pass parameters this way:

<jmxserver host="*****x6c8.******.it" jmxport="28580" jmxuser="us*****" jmxpass="*******">

I'm testing a 6.1 WAS. Port 28580 is Bootstrap port. When trying to connect to that endopoint, I get the following error:

"Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
    java.net.SocketTimeoutException: Read timed out]"

So I tried SOAP port (28581) but I get a worse error:

"Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]"

So, where did I go wrong?
Regards,
Stefano

UPDATES: seems like I was going partially wrong. I've talked to my WAS Admins and they've told me JMX are exposed on SOAP port. So I modified inputs.conf to use poll_mx4j_jmx.sh script instead, and in config.xml I specified protocol="soap".
so, in default/inputs.conf:

[script://./bin/poll_mx4j_jmx.sh config.xml]
interval = 60
sourcetype = jmx
index = jmx
disabled = 0

and in config.xml:

Now I have this different error:

2013-02-13 15:19:19,600 ERROR [Thread-1] host=uswasx6c8.usinet.it, jmxServiceURL=, jmxport=28581, jvmDescription=null, processID=0,systemErrorMessage="; nested exception is: 
    java.lang.NumberFormatException: For input string: """

Please help!

[Damien_Dallimor]

Assuming that JMX is setup correctly on the WAS6.1 side of things ,

...these are the instructions for connecting the Oracle JDK JConsole to WAS 6.1 , which will be the same steps required for SPLUNK for JMX.

Put this jar on the classpath (dump in SPLUNK4JMX/bin/lib )

com.ibm.ws.admin.client_6.1.0.jar

Put these jars on the boot classpath (dump in SPLUNK4JMX/bin/lib/boot and add a bootclasspath entry property in the poll_jmx startup script to load these jars )

ibmorbapi.jar
ibmorb.jar
ibmcfw.jar

So in the script you might add something like :

WAS_JARS_FOR_BOOTPATH=$LIB_DIR/boot/ibmorbapi.jar:$LIB_DIR/boot/ibmorb.jar:$LIB_DIR/boot/ibmcfw.jar   
BOOTPATH=$WAS_JARS_FOR_BOOTPATH   
"$JAVA" -Xbootclasspath/a:$BOOTPATH -Dsplunk4jmx.home=$SPLUNK4JMX_HOME $JAVA_OPTS -classpath $LIB_DIR:$POLLER_JARS $MAIN_CLASS $CONFIG_XML

Use this URL format in the SPLUNK4JMX config file in the jmxserver element's jmxServiceURL attribute :

service:jmx:iiop://<host>:<port>/jndi/JMXConnector

[stefano_guidoba]

I have modified poll_jmx.sh script and I pass to it the following attribute: -Dcom.ibm.CORBA.ConfigURL=file://$LIB_DIR/sas.client.props. I've taken sas.client.props file from my AS folder. Different error now:
02-26-2013 17:19:31.673 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" INFO: Loading mapping descriptors from jar:file:/opt/splunk/etc/apps/SPLUNK4JMX/bin/lib/jmxpoller.jar!/mapping.xml

Question: in SPLUNK4JMX user guide you mentioned you've run through an IIOP connection with an IBM J9. Do you have docs on that?
Thanks.

[stefano_guidoba]

I'm closer but am now experiencing security issues:
02-26-2013 16:29:32.710 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" JSAS1480I: Security is not enabled because the ConfigURL property file is not set.
02-26-2013 16:29:32.750 +0100 INFO ExecProcessor - Ran script: /opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml, took 2032.3 milliseconds to run, 0 bytes read.

I couldn't solve adding jmxuser and jmxpass parameters to config.xml. I guess I have to apply this: http://bit.ly/XY0K74

Did you need authentication when you tried?

[stefano_guidoba]

Update: I added to my BOOTPATH variable a couple of JARs taken from $WAS_HOME/runtimes folder, as suggested here: http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14625017. Now I'm getting a different error; here's the stacktrace:

02-26-2013 11:42:29.459 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" Feb 26, 2013 11:42:29 AM org.exolab.castor.mapping.Mapping loadMapping
02-26-2013 11:42:29.459 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" INFO: Loading mapping descriptors from jar:file:/opt/splunk/etc/apps/SPLUNK4JMX/bin/lib/jmxpoller.jar!/mapping.xml
02-26-2013 11:42:30.137 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" JSAS1480I: Security is not enabled because the ConfigURL property file is not set.
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" java.lang.NullPointerException
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.extension.thinregistry.RegistryLoader$1.run(RegistryLoader.java:58)
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at java.security.AccessController.doPrivileged(AccessController.java:229)
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.extension.thinregistry.RegistryLoader.addToRegistry(RegistryLoader.java:55)
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.extension.thinregistry.ExtensionRegistry.<init>(ExtensionRegistry.java:68)
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at java.lang.J9VMInternals.newInstanceImpl(Native Method)
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at java.lang.Class.newInstance(Class.java:1357)
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.extension.ExtensionRegistryFactoryImpl.<clinit>(ExtensionRegistryFactoryImpl.java:67)
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at java.lang.J9VMInternals.initializeImpl(Native Method)
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at java.lang.J9VMInternals.initialize(J9VMInternals.java:228)
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.wsspi.extension.ExtensionRegistryFactory.<clinit>(ExtensionRegistryFactory.java:53)
02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at java.lang.J9VMInternals.initializeImpl(Native Method)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at java.lang.J9VMInternals.initialize(J9VMInternals.java:228)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.util.ImplFactory.parseExtensionPoint(ImplFactory.java:198)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.util.ImplFactory.access$000(ImplFactory.java:81)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.util.ImplFactory$1.run(ImplFactory.java:108)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.util.ImplFactory.<clinit>(ImplFactory.java:105)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at java.lang.J9VMInternals.initializeImpl(Native Method)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at java.lang.J9VMInternals.initialize(J9VMInternals.java:228)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.util.PlatformHelperFactory.getPlatformHelper(PlatformHelperFactory.java:82)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.security.config.CSIv2ConfigImpl.initializeOnClient(CSIv2ConfigImpl.java:396)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.security.config.CSIv2ConfigImpl.init(CSIv2ConfigImpl.java:1123)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ISecurityUtilityImpl.SecurityPropertyManager.loadConfigURL(SecurityPropertyManager.java:652)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ISecurityUtilityImpl.SecurityPropertyManager.access$100(SecurityPropertyManager.java:87)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ISecurityUtilityImpl.SecurityPropertyManager$1.run(SecurityPropertyManager.java:155)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ISecurityUtilityImpl.SecurityPropertyManager.setInitialProperties(SecurityPropertyManager.java:153)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.CORBA.iiop.DefaultPropertyManager.init(DefaultPropertyManager.java:127)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.ISecurityUtilityImpl.SecurityPropertyManager.init(SecurityPropertyManager.java:694)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.rmi.corba.PluginRegistry.initPlugin(PluginRegistry.java:224)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.rmi.corba.PluginRegistry.initPlugins(PluginRegistry.java:210)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.rmi.corba.PluginRegistry.initPlugin(PluginRegistry.java:193)
02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.rmi.corba.ORB.initializePrereqPlugins(ORB.java:574)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.rmi.corba.ORB.initializePrereqPlugins(ORB.java:593)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.rmi.corba.ORB.orbParameters(ORB.java:1342)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1268)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1688)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at org.omg.CORBA.ORB.init(ORB.java:364)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at javax.management.remote.rmi.RMIConnector.resolveOrb(RMIConnector.java:1732)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1862)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:266)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:259)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:218)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.dtdsoftware.splunk.ProcessServerThread.connect(Unknown Source)
02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml"   at com.dtdsoftware.splunk.ProcessServerThread.run(Unknown Source)
02-26-2013 11:42:30.204 +0100 INFO  ExecProcessor - Ran script: /opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml, took 1319.8 milliseconds to run, 0 bytes read

Ideas?

[Damien_Dallimor]

Is port 28580 the correct JMX remote port ? You might see an entry in the WAS start script such as -Dcom.sun.management.jmxremote.port that you can reference to confirm.

Can you connect using JConsole ?

Do you have a firewall blocking the port ?

Also ,look over this thread , some additional JMX and WAS tips that might point you in the right direction.

http://stackoverflow.com/questions/357095/how-do-you-enable-jmx-in-websphere

[stefano_guidoba]

Oh, right! I'm now testing against a WAS 8.5, with a 1.6 JDK.

[stefano_guidoba]

I was able to connect using JConsole locally on server on which resides my IBM WAS and remotely, specifying only hostname:port parameters (no auth required).
I'm now making more tests with SPLUNK4JMX, still obtaining a ReadTimedOut Exception. I have no clues on what the problem might be..

[Damien_Dallimor]

I don't think you even have the JMX port setup correctly.That's what the error would suggest.You keep mentioned Bootstrap port , this is not the JMX server port.

What is you level of skill/knowledge with JMX ?

You need to setup JMX in WAS correctly in the first instance.This is mentioned in the Splunk4JMX documentation and in the link above.

The first thing I would suggest doing is trying to connect with JConsole. If you can achieve this, then you will then be able to connect with Splunk4JMX.

[stefano_guidoba]

Damien, be kind please: could you list the steps required to connect to an IBM WebSphere JVM? I'm going mad..

[stefano_guidoba]

2013-02-13 13:48:57,388 ERROR [Thread-1] host=uswasx6c8.usinet.it, jmxServiceURL=, jmxport=28580, jvmDescription=null, processID=0,systemErrorMessage="Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out]"

[stefano_guidoba]

Damien,
28580 is Bootstrap Port ( WAS default 2809) which your link points to if you want to gain JMX metrics. I have no firewalls between Splunk and this JVM (in fact, telnetting that port works fine). Also, I guess it's not a problem of credentials (at least, it would have shown a message with that info).
I'm getting a "Read Timeout" error.
I post the entire connection error event: