All Apps and Add-ons

Splunk JVM agent method enter/exit not working

New Member

Working on using Damien's JVM Instrumentation Agent mainly for the method tracing data. I have whitelisted my entire package structure and can see "classloaded" events in Splunk so I know it's connected to Splunk via TCP and the agent is running, but I do not get any methodentered and method_exited events as the app is being used.

Anybody else had experience with this?

Below is the relevant part of splunkagent.properties (hprof and jmx are disabled so not pasted here). I'm running in Weblogic 12.1.2, jdk1.7.0_51.

agent.app.name=myapp
agent.app.instance=myjvm

#-------------------------
# Splunk Transport options
#-------------------------
#splunk.transport.internalQueueSize=10000
splunk.transport.impl=com.splunk.javaagent.transport.SplunkTCPTransport
splunk.transport.tcp.host=splunk.mydomain.com
splunk.transport.tcp.port=40000
splunk.transport.tcp.maxQueueSize=5MB
splunk.transport.tcp.dropEventsOnQueueFull=false

#-----------------------------------
# Class/Method/Error Tracing options
#-----------------------------------
trace.whitelist=mypackage/
trace.methodEntered=true
trace.methodExited=true
trace.classLoaded=true
trace.errors=true
0 Karma

Ultra Champion

Are you using the latest version (1.0) of the Agent ?

0 Karma

New Member

I also fired up Tomcat 8, added the agent and deployed my sample 4 line test web service. Same results, classloaded entries, but nothing else. I removed the whitelist and got over 1k classloaded entries in Splunk from various Apache libraries, but still nothing else. I tried with 1.7.051 (x86) and 1.7.067 (x64), same results. Tried uncommenting the internalQueueSize and setting to 500000, no difference.

Below is my full file. I just pruned out your big set of HPROF comments.

Will ping you on email too.

#See the README for definition of the propertys

#---------------------
# Common Agent options
#---------------------

agent.app.name=myapp
agent.app.instance=myjvm
#agent.userEventTags=key1=value1,key2=value2

#-------------------------
# Splunk Transport options
#-------------------------
splunk.transport.internalQueueSize=500000
splunk.transport.impl=com.splunk.javaagent.transport.SplunkTCPTransport
#splunk.transport.impl=com.splunk.javaagent.transport.SplunkStdOutTransport
splunk.transport.tcp.host=splunk.mydomain.com
splunk.transport.tcp.port=40000
splunk.transport.tcp.maxQueueSize=5MB
splunk.transport.tcp.dropEventsOnQueueFull=false

#-----------------------------------
# Class/Method/Error Tracing options
#-----------------------------------
#trace.whitelist=mypackage/
#trace.blacklist=com/sun,sun/,java/,javax/,com/splunk/javaagent/
trace.methodEntered=true
trace.methodExited=true
trace.classLoaded=true
trace.errors=true


#------------------------------
# HPROF Dump Collection options
#------------------------------

#periodically dump hprof file(using JMX operation call), read in file & parse, send events to Splunk, delete file
trace.hprof=false
trace.hprof.tempfile=mydump.hprof
#trace.hprof.tempfile=/etc/tmp/dump.hprof
#in seconds
trace.hprof.frequency=30


#--------------------------------------------------------
# JMX attribute/operation/notification collection options
#--------------------------------------------------------

#Embedded JMX polling , all other JMX config is in the JMX XML configuration file.
trace.jmx=false
#name of XML files(minus the ".xml" suffix) that should reside in the root of splunkagent.jar
trace.jmx.configfiles=jmx
#trace.jmx.configfiles=goo,foo
#in seconds
trace.jmx.default.frequency=30
#trace.jmx.goo.frequency=30
0 Karma

Ultra Champion

I would not comment this out.

 #trace.blacklist=com/sun,sun/,java/,javax/,com/splunk/javaagent/

Also , can you please keep the conversation on the answers thread.This is community supported software , so I like to try and pool and knowledge from the help here vs other content getting lost in direct emails.

0 Karma

New Member

I only commented that blacklist out just to see if it was an issue with my package or not. I've tried all permutations (whitelist and blacklist uncommented, both commented, one or the other commented). Even with the blacklist commented out, I still only get the class_loaded entries even from the main Java packages.

Is it possible for you to drop your test somewhere I can get it?

0 Karma

Ultra Champion

1) download latest version of Tomcat 8 (http://tomcat.apache.org/download-80.cgi)

2) add agent to apache-tomcat-8.0.24/bin/catalina.sh line 98 (using properties file outside of jar)

CATALINA_OPTS="-javaagent:splunkagent.jar=/somepath/foo/splunkagent.properties"

3) start Tomcat 8

4) Search in Splunk , should see 100's thousands of events.

Versions:

Java version 1.7.0_67
Latest version of Splunk Java agent, v1.0
Mac OS 10.9.2
Splunk Version 6.2.4

My exact properties file content is : (you'll want to change the TCP host and port)

agent.app.name=tomcat8_test
agent.app.instance=jvm_1
#agent.userEventTags=key1=value1,key2=value2


splunk.transport.internalQueueSize=500000
splunk.transport.impl=com.splunk.javaagent.transport.SplunkTCPTransport
#splunk.transport.impl=com.splunk.javaagent.transport.SplunkStdOutTransport
splunk.transport.tcp.host=localhost
splunk.transport.tcp.port=5151
splunk.transport.tcp.maxQueueSize=5MB
splunk.transport.tcp.dropEventsOnQueueFull=false


#trace.whitelist=com/splunk/test
trace.blacklist=com/sun,sun/,java/,javax/,com/splunk/javaagent/
trace.methodEntered=false
trace.methodExited=false
trace.classLoaded=false
trace.errors=false


#Embedded JMX polling , all other JMX config is in the JMX XML configuration file.
trace.jmx=true
#name of XML files(minus the ".xml" suffix) that should reside in the root of splunkagent.jar
trace.jmx.configfiles=jmx
#trace.jmx.configfiles=goo,foo
#in seconds
trace.jmx.default.frequency=60
#trace.jmx.goo.frequency=30
0 Karma

New Member

Unfortunately downloading the latest did not seem to do anything. I still only see class_loaded events, no method entered or exited despite confirming setting to true to enable and running some traffic.

I even tried by just creating a new single JAX-WS hello web service (which does a few method calls to dummy classes) and instrumenting that, and again all I can see are class_loaded events.

I decompiled your jar and I see in SplunkJavaAgent, the classLoaded method uses agent.transport.send to send info directly, but all the others (like methodEntered) use agent.eventQueue.put (which in turn should send to agent.transport.send). Could there be something different going on there?

I took a thread dump and do see that TransporterThread running.

            "Thread-0" TIMED_WAITING          
            java.lang.Thread.sleep(Native Method)          
            com.splunk.javaagent.SplunkJavaAgent$TransporterThread.run(Unknown Source)

Is there a way to debug a java agent using Eclipse?

0 Karma

Ultra Champion

Can you post your full properties file ? Just so I can sanity check it.

Just a guess , perhaps Weblogic has some classloader hierarchy wierdness. By that I mean some JEE Appservers have sibling classloaders so I have some checks in the code to ensure that the class being instrumented can see the JavaAgent classes (to prevent ClassNotFoundException crashes). Are the classloaded events you see in Splunk for the classes in your whitelisted package ?

Also maybe try increasing the internal queue size

Uncomment the following option in splunkagent.properties :

#splunk.transport.internalQueueSize=10000

This for example sets the internal queue size to 500,000 events. When it fills up , it starts to block. The TCP transport handler sending to Splunk is the drain on the other side of the queue.

splunk.transport.internalQueueSize=500000

As a sanity check I just installed the latest Tomcat version 8 , wired up the agent , and everything streamed into Splunk just fine.

PS : if you email me , ddallimore@splunk.com , I can give you access to the git repo where you can view or fork the code.

0 Karma

New Member

I guess not. I'm using what looks like 0.5, the version linked from https://splunkbase.splunk.com/app/1716/ which points to http://damiendallimore.github.io/#javaagent

Where can I find the latest? Thank you!

0 Karma

Ultra Champion

Follow the "Download" link.

Release Notes are inside the tarball you download.

0 Karma