We need some guidance on how to configure Splunk to monitor IBM WebSphere Application Server and apps hosted on it.
We’ve installed
Splunk monitoring software on a Ubuntu VM.
Splunk Add-on for Java Management Extensions (JMX) (splunkbase.splunk.com/app/2647)
Splunk Add-on for IBM WebSphere Application Server (splunkbase.splunk.com/app/2789)
We could not configure add-on to connect to WebSphere. (docs.splunk.com/Documentation/AddOns/released/JMX/Configureinput)
Please guide us to configure splunk to monitor IBM WAS..
We do not know what values we should be entering on “Add Server” screen.
We tried following values for jmx_url:
service:jmx:iiop:///jndi/corbaname:iiop::9100/WsnAdminNameService#JMXConnector
service:jmx:rmi:///jndi/rmi://:9100/jmxrmi
service:jmx:iiop://10.55.68.60:2809/jndi/JMXConnector : This options results in following errors:
06-09-2017 17:43:07.286 +0530 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_internal~19~04FD6225-5ABD-4E92-A012-DA018F2F2434
06-09-2017 17:43:07.402 +0530 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'
06-09-2017 17:43:07.403 +0530 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db
06-09-2017 17:43:08.285 +0530 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_introspection~18~04FD6225-5ABD-4E92-A012-DA018F2F2434
06-09-2017 17:43:08.402 +0530 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'
06-09-2017 17:43:08.403 +0530 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db
06-09-2017 17:43:09.238 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" JSAS1480I: Security is not enabled because the ConfigURL property file is not set.
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" Exception in thread "Thread-2" java.lang.NoClassDefFoundError: sun/io/MalformedInputException
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.CDRReader.getTcsCConverter(CDRReader.java:451)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.CDRReader.readStringOrIndirection(CDRReader.java:532)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.CDRReader.read_string(CDRReader.java:518)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.IOR.read(IOR.java:337)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.Connection._locate(Connection.java:531)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.Connection.locate(Connection.java:490)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.GIOPImpl.locate(GIOPImpl.java:229)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.corba.Corbaloc.locateUsingINS(Corbaloc.java:307)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.corba.Corbaloc.resolve(Corbaloc.java:378)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.corba.ORB.objectURLToObject(ORB.java:3796)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.CORBA.iiop.ORB.objectURLToObject(ORB.java:3263)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.corba.ORB.string_to_object(ORB.java:3694)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtxFactory.stringToObject(WsnInitCtxFactory.java:1645)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtxFactory.getCosRootContext(WsnInitCtxFactory.java:1338)
06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtxFactory.getRootContextFromServer(WsnInitCtxFactory.java:1088)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtxFactory.getRootJndiContext(WsnInitCtxFactory.java:939)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtxFactory.getInitialContextInternal(WsnInitCtxFactory.java:614)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtx.getContext(WsnInitCtx.java:128)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtx.getContextIfNull(WsnInitCtx.java:765)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:164)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:179)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at javax.naming.InitialContext.lookup(InitialContext.java:417)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.websphere.management.remote.WsProvider.newJMXConnector(WsProvider.java:139)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at javax.management.remote.JMXConnectorFactory.getConnectorAsService(JMXConnectorFactory.java:476)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:344)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:269)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.splunk.jmx.ServerTask.connect(Unknown Source)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.splunk.jmx.ServerTask.checkConnector(Unknown Source)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.splunk.jmx.Scheduler.run(Unknown Source)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" Caused by: java.lang.ClassNotFoundException: sun.io.MalformedInputException
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" ... 29 more
06-09-2017 17:43:13.846 +0530 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/export_metrics.log'.
06-09-2017 17:43:34.009 +0530 INFO KeyManagerLocalhost - Checking for localhost key pair
06-09-2017 17:43:34.010 +0530 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-09-2017 17:43:34.010 +0530 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-09-2017 17:43:34.010 +0530 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
06-09-2017 17:43:34.010 +0530 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
06-09-2017 17:43:34.010 +0530 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
06-09-2017 17:44:04.404 +0530 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in
... View more