Just putting a simple idea out there on how to query the jmxproxy. If you think that is convoluted, wait until you read the rest. Should be enough to drive people into the arms of your app. This information is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. And definitely WITHOUT ANY REGARD FOR SECURITY or scalability. Lets try to make a Dashboard and panel for Tomcat heap memory usage for the past 60 minutes. I used Splunk 5.0 for this bit. SPLUNK_HOME is set to /opt/splunk and Splunk runs under a user named splunk in my environment. Make the directory for scripted input. mkdir -p $SPLUNK_HOME/etc/apps/curl/bin Create a script as follows named curl.sh in the $SPLUNK_HOME/etc/apps/curl/bin directory. !/bin/bash PATH=/bin:/usr/bin curl "$@" | sed -r -e 's/:\s+?/=/' -e 's/\r$//' | xargs echo -n --- end of script --- Set ownership and permissions. chown splunk $SPLUNK_HOME/etc/apps/curl/bin/curl.sh chmod 750 $SPLUNK_HOME/etc/apps/curl/bin/curl.sh Test it out manually to see if it works as expected. $SPLUNK_HOME/etc/apps/curl/bin/curl.sh -s --user tomcat:tomcat 'http://buildc6x86_64:8080/manager/jmxproxy/?qry=java.lang:type=Memory' Resulting output as expected from manual test. OK - Number of results=1 Name=java.lang:type=Memory modelerType=sun.management.MemoryImpl Verbose=false HeapMemoryUsage=javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.MemoryUsage,items=((itemName=committed,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=init,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=max,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=used,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={committed=259522560, init=268435456, max=259522560, used=64526256}) NonHeapMemoryUsage=javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.MemoryUsage,items=((itemName=committed,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=init,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=max,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=used,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={committed=24313856, init=24313856, max=136314880, used=19639688}) ObjectPendingFinalizationCount=0 ObjectName=java.lang:type=Memory Let's use Splunk's web console to add a Data input under Splunk> Manager >> Data input >> Script >> Add new. Command: /opt/splunk/etc/apps/curl/bin/curl.sh -s --user tomcat:tomcat 'http://buildc6x86_64:8080/manager/jmxproxy/?qry=java.lang:type=Memory' Interval: 60 Source name override: curl_jmxproxy_tomcat_memory Set sourcetype: Manual Source type: curl_jmxproxy_tomcat_memory Click on Save. Go to the Search App and input this search. Select "Last 60 minutes" to execute the search. We are using rex to extract fields. source="curl_jmxproxy_tomcat_memory" host="buildc6x86_64" | rex "(?i).+?\s+HeapMemoryUsage=.+,contents={committed=(?P<HC>\d+),\s+init=(?P<HI>\d+),\s+max=(?P<HM>\d+),\s+used=(?P<HU>\d+)}.+\s+NonHeapMemoryUsage=" | eval HeapCommitted=HC/1048576 | eval HeapUsed=HU/1048576 | eval HeapFree=HeapCommitted-HeapUsed | timechart span=1m avg(HeapFree) avg(HeapUsed) Click on Save > Save search... Name: Tomcat memory usage last 60 minutes Click on Dashboard & Views > Create dashboard... ID: tomcat_stats Name: Tomcat Stats Click on Save. Enable Edit mode (On) in the Tomcat Stats dashboard. Click on + new panel. Title: Tomcat memory usage last 60 minutes Saved search (drop down menu): Tomcat memory usage last 60 minutes Click on Save. Enable Edit > Edit visualization in the Tomcat memory usage last 60 minutes panel. Select Area in the Visualizations drop down menu. Under General, select the second stack mode icon. Select X Axis and type Time in the X Axis title field. Select Y Axis and type Megabytes in the Y Axis title field. Click on Save. Disable Edit mode (Off) in the Tomcat Stats dashboard. The end. ... View more

Tomcat provides jmxproxy which is similar to the jmx-console provided by JBoss. Be aware that there is line wrapping with the commands I posted below. I tested this against tomcat-7.0.25 running on a CentOS 6.3 x86_64 system. For tomcat, I enabled the manager-jmx role in the .../conf/tomcat-users.xml file and restarted tomcat. It should read similar to the following: <tomcat-users> <role rolename="manager-jmx"/> <user username="tomcat" password="tomcat" roles="manager-jmx"/> </tomcat-users> Now I can use curl to query the java.lang mbean for all the types: curl -s --user tomcat:tomcat 'http://buildc6x86_64:8080/manager/jmxproxy/?qry=java.lang:type=*' | grep ^Name: Name: java.lang:type=Compilation Name: HotSpot 64-Bit Tiered Compilers Name: java.lang:type=Memory Name: java.lang:type=Runtime Name: 11667@buildc6x86_64.example.com Name: java.lang:type=ClassLoading Name: java.lang:type=Threading Name: java.lang:type=OperatingSystem Name: Linux Query the Threading type: curl -s --user tomcat:tomcat 'http://buildc6x86_64:8080/manager/jmxproxy/?qry=java.lang:type=Threading' OK - Number of results: 1 Name: java.lang:type=Threading modelerType: sun.management.ThreadImpl ThreadAllocatedMemoryEnabled: true ThreadAllocatedMemorySupported: true ThreadContentionMonitoringEnabled: false DaemonThreadCount: 22 PeakThreadCount: 23 CurrentThreadCpuTimeSupported: true ObjectMonitorUsageSupported: true SynchronizerUsageSupported: true ThreadContentionMonitoringSupported: true ThreadCpuTimeEnabled: true AllThreadIds: Array[long] of length 23 31 30 29 28 27 26 25 24 23 22 20 19 18 17 16 13 12 11 10 4 3 2 1 CurrentThreadCpuTime: 44567045 CurrentThreadUserTime: 30000000 ThreadCount: 23 TotalStartedThreadCount: 25 ThreadCpuTimeSupported: true ObjectName: java.lang:type=Threading Change the query results by replacing the first colon (:) with an equals sign (=), removing carriage returns from end of line, and turning the output into one line for easier parsing in Splunk. curl -s --user tomcat:tomcat 'http://buildc6x86_64:8080/manager/jmxproxy/?qry=java.lang:type=Threading' | sed -r -e 's/:\s+?/=/' -e 's/\r$//' | xargs echo -n OK - Number of results=1 Name=java.lang:type=Threading modelerType=sun.management.ThreadImpl ThreadAllocatedMemoryEnabled=true ThreadAllocatedMemorySupported=true ThreadContentionMonitoringEnabled=false DaemonThreadCount=22 PeakThreadCount=23 CurrentThreadCpuTimeSupported=true ObjectMonitorUsageSupported=true SynchronizerUsageSupported=true ThreadContentionMonitoringSupported=true ThreadCpuTimeEnabled=true AllThreadIds=Array[long] of length 23 31 30 29 28 27 26 25 24 23 22 20 19 18 17 16 13 12 11 10 4 3 2 1 CurrentThreadCpuTime=35302227 CurrentThreadUserTime=20000000 ThreadCount=23 TotalStartedThreadCount=25 ThreadCpuTimeSupported=true ObjectName=java.lang:type=Threading At this point, you have the basis for a wrapper script to be used as scripted input by Splunk. Outputs that aren't cleaned up enough to show up as fields in Splunk can be parsed further in a number of ways; in your wrapper script, using transforms in Splunk, within Splunk using PCRE... I used jconsole to browse some of values given above, but that is another story. All this info is on the 'net of course. I am going to rewrite using Perl and make things nicer for myself, but now you are fishing. ... View more