Unable to get JMX data with app Monitoring of Java Virtual Machines with JMX

[DFresh4130]

I’ve been working on installing Splunk for JMX app for a while now with no luck. Maybe I’m doing something wrong. I’ve performed the following steps and not having any luck seeing data.

• Splunk 6.0.1
• Followed instructions to untar in the apps directory
• Restarted Splunk
• Now seeing the app inside the web interface

This is where I'm stuck. The instructions say “You can configure your JMX inputs via Manager->DataInputs->JMX”, but I don’t see JMX under the data inputs. I even went onto the server and modified the config.xml with a valid host like below:

<jmxserver host="host1" jvmDescription="APP1_Tomcat1" jmxport="8880">

Also seeing the below entry in the logs on startup.

08-21-2014 14:04:07.270 -0400 INFO SpecFiles - Found external scheme definition for stanza "jmx://" with 3 parameters: config_file, config_file_dir, polling_frequency
08-21-2014 14:04:07.270 -0400 INFO SpecFiles - Found external scheme definition for stanza "perfmon://" with 10 parameters: object, counters, instances, interval, mode, samplingInterval, stats, disabled, index, showZeroValue
08-21-2014 14:04:08.285 -0400 ERROR ModularInputs - Introspecting scheme=jmx: script running failed (exited with code 1).
08-21-2014 14:04:08.285 -0400 ERROR ModularInputs - Unable to initialize modular input "jmx" defined inside the app "jmx_ta": Introspecting scheme=jmx: script running failed (exited with code 1).
08-21-2014 14:04:08.286 -0400 INFO ClusteringMgr - initing clustering with: ht=60 rf=3 sf=2 ct=60 st=60 rt=60 rct=60 rst=60 rrt=60 rmst=600 rmrt=600 sfrt=600 pe=1 im=0 is=0 mob=2 mor=5 pb=5 rep_port= pptr=10


I've validated my splunk server has connectivity to the tomcat JMX port and I can connect to it through visualVM with no problems. JMX security is turned off too. Any thoughts on where I need to look? Thanks

UPDATE
This is an old Oracle linux 5.6 server. The latest version of java in the repos for it is 1.4.2. Is it possible that splunk is trying to use /usr/bin/java instead of what I have defined with JAVA_HOME?

UPDATE 2
I found that /usr/bin/java was really just a symlink instead of the real binary. I went ahead and updated it to point to the 1.7 JDK path and now the app appears to be coming up normally.

08-21-2014 17:48:34.647 -0400 INFO SpecFiles - Found external scheme definition for stanza "jmx://" with 3 parameters: config_file, config_file_dir, polling_frequency
08-21-2014 17:48:34.647 -0400 INFO SpecFiles - Found external scheme definition for stanza "perfmon://" with 10 parameters: object, counters, instances, interval, mode, samplingInterval, sta
ts, disabled, index, showZeroValue
08-21-2014 17:48:36.997 -0400 INFO ModularInputs - Endpoint argument settings for "config_file":
08-21-2014 17:48:36.997 -0400 INFO ModularInputs - Endpoint argument settings for "config_file_dir":
08-21-2014 17:48:36.997 -0400 INFO ModularInputs - Endpoint argument settings for "name":
08-21-2014 17:48:36.997 -0400 INFO ModularInputs - Endpoint argument settings for "polling_frequency":
08-21-2014 17:48:36.997 -0400 INFO ModularInputs - Introspection setup completed for scheme "jmx".
08-21-2014 17:48:36.998 -0400 INFO ClusteringMgr - initing clustering with: ht=60 rf=3 sf=2 ct=60 st=60 rt=60 rct=60 rst=60 rrt=60 rmst=600 rmrt=600 sfrt=600 pe=1 im=0 is=0 mob=2 mor=5 pb=5
rep_port= pptr=10
08-21-2014 17:48:37.516 -0400 INFO HotDBManager - idx=jmx Setting hot mgr params: maxHotSpanSecs=7776000 snapBucketTimespans=false maxHotBuckets=3 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
08-21-2014 17:48:37.516 -0400 INFO databasePartitionPolicy - idx=jmx Initialized, params='[300,60,188697600,,,,786432000,5,true,500000,5,5,false,6,0,_blocksignature,7776000,1000000,0,3,77760000,2592000,131072,25,0,15,0,0,-1,18446744073709551615,2592000,true,60000,300000,false,2000]' isSlave=false needApplyDeleteJournal=false
08-21-2014 17:48:37.516 -0400 INFO HotDBManager - closing hot mgr for idx=jmx
08-21-2014 17:48:37.517 -0400 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/splunk/var/lib/splunk/jmx/db'. Reason='Refreshing manifest.'
08-21-2014 17:48:37.519 -0400 INFO databasePartitionPolicy - openDatabases complete currentId=54 idx=jmx
08-21-2014 17:48:37.780 -0400 INFO ModularInputs - No stanzas found for scheme "jmx" in inputs.conf at script (re)start.
08-21-2014 17:48:37.781 -0400 INFO ExecProcessor - New scheduled exec process: python /splunk/etc/apps/jmx_ta/bin/jmx.py
08-21-2014 17:48:53.783 -0400 ERROR ExecProcessor - message from "python /splunk/etc/apps/jmx_ta/bin/jmx.py" It has been determined via the REST API that all inputs have been disabled

Now it appears I have another configuration issue. Do I need to update more than just the example config.xml with a host and jmx port to get data coming in correctly? I'm able to see the JMX data inputs in the console now with helloworld pointing to config.xml.

[DFresh4130 · Accepted solution]

Ok, first problem was fixed by updating the /usr/bin/java symlink to point it to a supported version. Not sure why JAVA_HOME variable wasn't working. Second issue was having to enable the JMX input in the console. Since the JMX app wasn't working properly it was disabled by default.

[DFresh4130]

Yes, I do have the jdk installed and JAVA_HOME set.

[splunk]# echo $JAVA_HOME
/apps/jdk1.7.0_02

[splunk]# /apps/jdk1.7.0_02/bin/java -version
java version "1.7.0_02"
Java(TM) SE Runtime Environment (build 1.7.0_02-b13)
Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode)

[Damien_Dallimor]

The Mod Input is not even loading , which suggests something core that you have overlooked.

Do you have a Java Runtime installed ?