How to configure Splunk to monitor IBM WebSphere Application Server?

[sanjayr]

We need some guidance on how to configure Splunk to monitor IBM WebSphere Application Server and apps hosted on it.

We’ve installed

1. Splunk monitoring software on a Ubuntu VM.
2. Splunk Add-on for Java Management Extensions (JMX) (splunkbase.splunk.com/app/2647)
3. Splunk Add-on for IBM WebSphere Application Server (splunkbase.splunk.com/app/2789)

We could not configure add-on to connect to WebSphere. (docs.splunk.com/Documentation/AddOns/released/JMX/Configureinput)
Please guide us to configure splunk to monitor IBM WAS..

We do not know what values we should be entering on “Add Server” screen.
We tried following values for jmx_url:

1. service:jmx:iiop:///jndi/corbaname:iiop::9100/WsnAdminNameService#JMXConnector
2. service:jmx:rmi:///jndi/rmi://:9100/jmxrmi
3. service:jmx:iiop://10.55.68.60:2809/jndi/JMXConnector : This options results in following errors: 06-09-2017 17:43:07.286 +0530 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_internal~19~04FD6225-5ABD-4E92-A012-DA018F2F2434 06-09-2017 17:43:07.402 +0530 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1' 06-09-2017 17:43:07.403 +0530 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db 06-09-2017 17:43:08.285 +0530 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_introspection~18~04FD6225-5ABD-4E92-A012-DA018F2F2434 06-09-2017 17:43:08.402 +0530 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1' 06-09-2017 17:43:08.403 +0530 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db 06-09-2017 17:43:09.238 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" JSAS1480I: Security is not enabled because the ConfigURL property file is not set. 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" Exception in thread "Thread-2" java.lang.NoClassDefFoundError: sun/io/MalformedInputException 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.CDRReader.getTcsCConverter(CDRReader.java:451) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.CDRReader.readStringOrIndirection(CDRReader.java:532) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.CDRReader.read_string(CDRReader.java:518) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.IOR.read(IOR.java:337) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.Connection._locate(Connection.java:531) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.Connection.locate(Connection.java:490) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.iiop.GIOPImpl.locate(GIOPImpl.java:229) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.corba.Corbaloc.locateUsingINS(Corbaloc.java:307) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.corba.Corbaloc.resolve(Corbaloc.java:378) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.corba.ORB.objectURLToObject(ORB.java:3796) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.CORBA.iiop.ORB.objectURLToObject(ORB.java:3263) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.rmi.corba.ORB.string_to_object(ORB.java:3694) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtxFactory.stringToObject(WsnInitCtxFactory.java:1645) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtxFactory.getCosRootContext(WsnInitCtxFactory.java:1338) 06-09-2017 17:43:09.508 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtxFactory.getRootContextFromServer(WsnInitCtxFactory.java:1088) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtxFactory.getRootJndiContext(WsnInitCtxFactory.java:939) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtxFactory.getInitialContextInternal(WsnInitCtxFactory.java:614) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtx.getContext(WsnInitCtx.java:128) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtx.getContextIfNull(WsnInitCtx.java:765) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:164) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:179) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at javax.naming.InitialContext.lookup(InitialContext.java:417) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.ibm.websphere.management.remote.WsProvider.newJMXConnector(WsProvider.java:139) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at javax.management.remote.JMXConnectorFactory.getConnectorAsService(JMXConnectorFactory.java:476) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:344) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:269) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.splunk.jmx.ServerTask.connect(Unknown Source) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.splunk.jmx.ServerTask.checkConnector(Unknown Source) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at com.splunk.jmx.Scheduler.run(Unknown Source) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" Caused by: java.lang.ClassNotFoundException: sun.io.MalformedInputException 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 06-09-2017 17:43:09.509 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" ... 29 more 06-09-2017 17:43:13.846 +0530 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/export_metrics.log'. 06-09-2017 17:43:34.009 +0530 INFO KeyManagerLocalhost - Checking for localhost key pair 06-09-2017 17:43:34.010 +0530 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem 06-09-2017 17:43:34.010 +0530 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem 06-09-2017 17:43:34.010 +0530 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem 06-09-2017 17:43:34.010 +0530 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem 06-09-2017 17:43:34.010 +0530 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem 06-09-2017 17:44:04.404 +0530 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in

[DalJeanis]

Verify that you have the IBM JRE installed that is appropriate to your CPU. You do not need the whole JDK (Java Development Kit), just the JRE (Java Runtime Environment).

For Linux and various other OSes on various CPUs, that is here...

https://developer.ibm.com/javasdk/downloads/sdk8/

That page also has links to send you here for Windows...

https://developer.ibm.com/javasdk/downloads/eclipse/

[raj_mpl]

Hi @Sanjayr could you please provide some information here ,
1.Have you used any apps to develop the dashboards for the application which is installed in websphere application server?
2.If the application is hosted in WAS , guide me how to connect it with splunk , log format is SystemOut.log and required dashboards ?

[DalJeanis]

@raj_mpl - asking questions of your own on an unanswered year-old question is highly unlikely to get you any useful advice. Please submit your own, carefully worded question, showing what you have tried, and what your specific issue is.