App Wanted: Tomcat

[Justin_Grant]

There's an app on Splunkbase for WebSphere, but is anyone building a similar app for Tomcat?

[arorarahul118]

I am using Jmx Add-on need help to create custom template to get catalina as mbean domain?

[amtowsbnp]

Just putting a simple idea out there on how to query the jmxproxy. If you think that is convoluted, wait until you read the rest. Should be enough to drive people into the arms of your app.

This information is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. And definitely WITHOUT ANY REGARD FOR SECURITY or scalability.

Lets try to make a Dashboard and panel for Tomcat heap memory usage for the past 60 minutes. I used Splunk 5.0 for this bit.

SPLUNK_HOME is set to /opt/splunk and Splunk runs under a user named splunk in my environment.

Make the directory for scripted input.

mkdir -p $SPLUNK_HOME/etc/apps/curl/bin

Create a script as follows named curl.sh in the $SPLUNK_HOME/etc/apps/curl/bin directory.

!/bin/bash

PATH=/bin:/usr/bin

curl "$@" | sed -r -e 's/:\s+?/=/' -e 's/\r$//' | xargs echo -n

--- end of script ---

Set ownership and permissions.

chown splunk $SPLUNK_HOME/etc/apps/curl/bin/curl.sh

chmod 750 $SPLUNK_HOME/etc/apps/curl/bin/curl.sh

Test it out manually to see if it works as expected.

$SPLUNK_HOME/etc/apps/curl/bin/curl.sh -s --user tomcat:tomcat 'http://buildc6x86_64:8080/manager/jmxproxy/?qry=java.lang:type=Memory'

Resulting output as expected from manual test.

OK - Number of results=1 Name=java.lang:type=Memory modelerType=sun.management.MemoryImpl Verbose=false HeapMemoryUsage=javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.MemoryUsage,items=((itemName=committed,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=init,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=max,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=used,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={committed=259522560, init=268435456, max=259522560, used=64526256}) NonHeapMemoryUsage=javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.MemoryUsage,items=((itemName=committed,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=init,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=max,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=used,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={committed=24313856, init=24313856, max=136314880, used=19639688}) ObjectPendingFinalizationCount=0 ObjectName=java.lang:type=Memory

Let's use Splunk's web console to add a Data input under Splunk> Manager >> Data input >> Script >> Add new.

Command: /opt/splunk/etc/apps/curl/bin/curl.sh -s --user tomcat:tomcat 'http://buildc6x86_64:8080/manager/jmxproxy/?qry=java.lang:type=Memory'

Interval: 60
Source name override: curl_jmxproxy_tomcat_memory
Set sourcetype: Manual
Source type: curl_jmxproxy_tomcat_memory

Click on Save.

Go to the Search App and input this search. Select "Last 60 minutes" to execute the search. We are using rex to extract fields.

source="curl_jmxproxy_tomcat_memory" host="buildc6x86_64" | rex "(?i).+?\s+HeapMemoryUsage=.+,contents={committed=(?P<HC>\d+),\s+init=(?P<HI>\d+),\s+max=(?P<HM>\d+),\s+used=(?P<HU>\d+)}.+\s+NonHeapMemoryUsage=" | eval HeapCommitted=HC/1048576 | eval HeapUsed=HU/1048576 | eval HeapFree=HeapCommitted-HeapUsed | timechart span=1m avg(HeapFree) avg(HeapUsed)

Click on Save > Save search...

Name: Tomcat memory usage last 60 minutes

Click on Dashboard & Views > Create dashboard...

ID: tomcat_stats

Name: Tomcat Stats

Click on Save.

Enable Edit mode (On) in the Tomcat Stats dashboard.
Click on + new panel.

Title: Tomcat memory usage last 60 minutes

Saved search (drop down menu): Tomcat memory usage last 60 minutes

Click on Save.

Enable Edit > Edit visualization in the Tomcat memory usage last 60 minutes panel.

Select Area in the Visualizations drop down menu.

Under General, select the second stack mode icon.

Select X Axis and type Time in the X Axis title field.

Select Y Axis and type Megabytes in the Y Axis title field.

Click on Save.

Disable Edit mode (Off) in the Tomcat Stats dashboard.

The end.

[amtowsbnp]

Tomcat provides jmxproxy which is similar to the jmx-console provided by JBoss. Be aware that there is line wrapping with the commands I posted below. I tested this against tomcat-7.0.25 running on a CentOS 6.3 x86_64 system.

For tomcat, I enabled the manager-jmx role in the .../conf/tomcat-users.xml file and restarted tomcat. It should read similar to the following:

<tomcat-users>

<role rolename="manager-jmx"/>
<user username="tomcat" password="tomcat" roles="manager-jmx"/>

</tomcat-users>

Now I can use curl to query the java.lang mbean for all the types:

curl -s --user tomcat:tomcat 'http://buildc6x86_64:8080/manager/jmxproxy/?qry=java.lang:type=*' | grep ^Name:

Name: java.lang:type=Compilation
Name: HotSpot 64-Bit Tiered Compilers
Name: java.lang:type=Memory
Name: java.lang:type=Runtime
Name: 11667@buildc6x86_64.example.com
Name: java.lang:type=ClassLoading
Name: java.lang:type=Threading
Name: java.lang:type=OperatingSystem
Name: Linux

Query the Threading type:

curl -s --user tomcat:tomcat 'http://buildc6x86_64:8080/manager/jmxproxy/?qry=java.lang:type=Threading'

OK - Number of results: 1

Name: java.lang:type=Threading
modelerType: sun.management.ThreadImpl
ThreadAllocatedMemoryEnabled: true
ThreadAllocatedMemorySupported: true
ThreadContentionMonitoringEnabled: false
DaemonThreadCount: 22
PeakThreadCount: 23
CurrentThreadCpuTimeSupported: true
ObjectMonitorUsageSupported: true
SynchronizerUsageSupported: true
ThreadContentionMonitoringSupported: true
ThreadCpuTimeEnabled: true
AllThreadIds: Array[long] of length 23
31
30
29
28
27
26
25
24
23
22
20
19
18
17
16
13
12
11
10
4
3
2
1
CurrentThreadCpuTime: 44567045
CurrentThreadUserTime: 30000000
ThreadCount: 23
TotalStartedThreadCount: 25
ThreadCpuTimeSupported: true
ObjectName: java.lang:type=Threading

Change the query results by replacing the first colon (:) with an equals sign (=), removing carriage returns from end of line, and turning the output into one line for easier parsing in Splunk.

curl -s --user tomcat:tomcat 'http://buildc6x86_64:8080/manager/jmxproxy/?qry=java.lang:type=Threading' | sed -r -e 's/:\s+?/=/' -e 's/\r$//' | xargs echo -n

OK - Number of results=1 Name=java.lang:type=Threading modelerType=sun.management.ThreadImpl ThreadAllocatedMemoryEnabled=true ThreadAllocatedMemorySupported=true ThreadContentionMonitoringEnabled=false DaemonThreadCount=22 PeakThreadCount=23 CurrentThreadCpuTimeSupported=true ObjectMonitorUsageSupported=true SynchronizerUsageSupported=true ThreadContentionMonitoringSupported=true ThreadCpuTimeEnabled=true AllThreadIds=Array[long] of length 23 31 30 29 28 27 26 25 24 23 22 20 19 18 17 16 13 12 11 10 4 3 2 1 CurrentThreadCpuTime=35302227 CurrentThreadUserTime=20000000 ThreadCount=23 TotalStartedThreadCount=25 ThreadCpuTimeSupported=true ObjectName=java.lang:type=Threading

At this point, you have the basis for a wrapper script to be used as scripted input by Splunk. Outputs that aren't cleaned up enough to show up as fields in Splunk can be parsed further in a number of ways; in your wrapper script, using transforms in Splunk, within Splunk using PCRE...

I used jconsole to browse some of values given above, but that is another story. All this info is on the 'net of course. I am going to rewrite using Perl and make things nicer for myself, but now you are fishing.

[Damien_Dallimor]

That is a rather convoluted approach.

Using "Splunk for JMX" makes this alot easier and simpler and data is already output to Splunk in best practice semantic format. See my configuration example for Tomcat earlier in this thread.

[muddu4you]

no app for tomcat yet ? bad!

[Damien_Dallimor]

I have a few "JVM" Splunk apps in the pipeline that augment the functionality of Splunk for JMX. Tomcat is one such app.

In the interim you can use Splunk for JMX and point it to the "java.lang" and "Catalina" MBean domains, there is a mass of useful JMX attributes and operations to chew over. I'm currently doing this for some large Tomcat deployments.

Here is a simple example config file to get you started.

<jmxpoller>

<!-- custom splunk output formatter-->

<formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameQuotesStrippedFormatter" />

<cluster name="TomcatInstances" description="All My Tomcat Servers">

<!-- 
A really simple example, this just discovers and dumps ALL MBean attributes in the "Catalina" and "java.lang" domains 
You'll want to figure out what attributes and operations you are specifically interested in and declare them individually.
-->

<mbean domain="Catalina" properties="*" dumpAllAttributes="true" />
<mbean domain="java.lang" properties="*" dumpAllAttributes="true" />


<!--There are many different ways to connect to the JMX server, read the docs.-->

<jmxserver host="host1" jvmDescription="Tomcat1" jmxport="9000"/>
<jmxserver host="host2" jvmDescription="Tomcat2" jmxport="9000"/>
<jmxserver host="host3" jvmDescription="Tomcat3" jmxport="9000"/>
<jmxserver host="host4" jvmDescription="Tomcat4" jmxport="9000"/>

</cluster>

</jmxpoller>

Read the PDF docs first, but feel free to drop me a line if you want any help with a Splunk for JMX config file setup for Tomcat.

[mdkellynzqa]

Any progress?

[rroberts]

Tomcat app would be useful

[cfergus]

We're gonna take a stab at it. Maybe within a month we can get something on Splunkbase.