- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?
JVM Instrumentation Agent is a convenient monitoring tool for the community. But I could not find any document to configure splunk ui to retrieve the raw apm event streamed data in enterprise UI. The agent properties are well defined and documented, though.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
DD, please also check following thread dump. It seems splunk agent blocked the thread (maybe in deadlock?):
"Processor-Thread-0" prio=5 tid=0x00007f8df9169000 nid=0x6307 waiting on condition [0x000000011d08a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000700186c78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:324)
at com.splunk.javaagent.SplunkJavaAgent.methodEntered(Unknown Source)
at com.xxx.Processor.getCookie(Processor.java)
at com.xxx.Processor.verify( Processor.java:216)
at com.xxx.Processor$1.run( Processor.java:79)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
DD, good catch!
Now I got VerifyException due to the agent which is causing the main app not booting up. I added -noverify jvm argument so booting up was fine but the method should be invoked was hanging. if I dont attach the agent, everything works fine. it seems there are incompatibility between jmx and the agent. any suggestion?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

I don't understand why your are bringing up JMX now.
Is this a typo , did you mean JVM ?
If so , What Java runtime version are your using ? 6,7,8 ?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

Your whitelist notation is wrong.
In my examples and the documentation I use "/" not "." as the package path seperator
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi DD,
thanks for quick reply. I manually defined the sourcetype "splunk_java_agent" and "tcp_raw" as indicated but none of them works. Is there any extra step missed to configure "splunk_java_agent"? from your slides, I know this sourcetype should work.
here is the main app jvm splunkagent.properties. the main modifications from your default file is the host, port and whitelist
See the README for definition of the propertys
---------------------
Common Agent options
---------------------
agent.app.name=verifier
agent.app.instance=verifierJVM1
agent.userEventTags=key1=value1,key2=value2
-------------------------
Splunk Transport options
-------------------------
splunk.transport.internalQueueSize=10000
splunk.transport.impl=com.splunk.javaagent.transport.SplunkTCPTransport
splunk.transport.impl=com.splunk.javaagent.transport.SplunkStdOutTransport
splunk.transport.tcp.host=172.16.3.1
splunk.transport.tcp.port=5250
splunk.transport.tcp.maxQueueSize=5MB
splunk.transport.tcp.dropEventsOnQueueFull=false
-----------------------------------
Class/Method/Error Tracing options
-----------------------------------
trace.whitelist=com.tealium.tiqverifier.selenium.UtagVerifyDataProcessor:verify,com.tealium.rest.endpoints.tiqverifier.VerifierConfigurationRestResource:runTestConfig
trace.blacklist=com/sun,sun/,java/,javax/,com/splunk/javaagent/
trace.methodEntered=true
trace.methodExited=true
trace.classLoaded=true
trace.errors=true
------------------------------
HPROF Dump Collection options
------------------------------
periodically dump hprof file(using JMX operation call), read in file & parse, send events to Splunk, delete file
trace.hprof=true
trace.hprof.tempfile=mydump.hprof
trace.hprof.tempfile=/etc/tmp/dump.hprof
in seconds
trace.hprof.frequency=30
trace.hprof.recordtypes=2,3,4
by default , all hprof records will be traced, but you can provide a comma delimited list of
specific record types to trace, the numbers are decimal values from the hprof spec.
HPROF RECORD TYPE CODE
==================================
HPROF_UTF8 1
HPROF_LOAD_CLASS 2
HPROF_UNLOAD_CLASS 3
HPROF_FRAME 4
HPROF_TRACE 5
HPROF_ALLOC_SITES 6
HPROF_HEAP_SUMMARY 7
HPROF_START_THREAD 10
HPROF_END_THREAD 11
HPROF_HEAP_DUMP 12
HPROF_HEAP_DUMP_SEGMENT 28
HPROF_HEAP_DUMP_END 44
HPROF_CPU_SAMPLES 13
HPROF_CONTROL_SETTINGS 14
Heap dumps(codes 12 & 28) can have subrecords , so you can use ":" notation to reference these values
trace.hprof.recordtypes=12:1,12:32,12:33,12:34
HEAPDUMP SUB RECORD TYPE CODE
=========================================
SUBRECORD_GC_ROOT_UNKNOWN -1
SUBRECORD_GC_ROOT_JNI_GLOBAL 1
SUBRECORD_GC_ROOT_JNI_LOCAL 2
SUBRECORD_GC_ROOT_JAVA_FRAME 3
SUBRECORD_GC_ROOT_NATIVE_STACK 4
SUBRECORD_GC_ROOT_STICKY_CLASS 5
SUBRECORD_GC_ROOT_THREAD_BLOCK 6
SUBRECORD_GC_ROOT_MONITOR_USED 7
SUBRECORD_GC_ROOT_THREAD_OBJ 8
SUBRECORD_GC_CLASS_DUMP 32
SUBRECORD_GC_INSTANCE_DUMP 33
SUBRECORD_GC_OBJ_ARRAY_DUMP 34
SUBRECORD_GC_PRIM_ARRAY_DUMP 35
--------------------------------------------------------
JMX attribute/operation/notification collection options
--------------------------------------------------------
Embedded JMX polling , all other JMX config is in the JMX XML configuration file.
trace.jmx=true
name of XML files(minus the ".xml" suffix) that should reside in the root of splunkagent.jar
trace.jmx.configfiles=jmx
trace.jmx.configfiles=goo,foo
in seconds
trace.jmx.default.frequency=30
trace.jmx.goo.frequency=30
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

You should not use the log4j sourcetype.
You should define your own sourcetype when you setup the TCP input.
Can you post your splunkagent.properties files.
Perhaps you have an error.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi DD, could you please help to identify my issue? here is what I did but not working:
the use case: two JVMs, one with Jetty and another with main application; both applied unique properties file with dedicated port # (e.g., 5150 for jetty agent and 5250 for main app) and whitelist from your provided
in Splunk Enterprise UI, I created two TCP configurations under "Local Inputs" as below. I could not figure out which source type I should use but copied your slides. the only worked source type is log4j (only for jetty jvm) while the whitelist does not work (it pull everything not only the classes/methods I specified in whitelist).
TCP port Host Restriction Source type Status Actions
5150 splunk_java_agent Enabled Clone | Delete
5250 tcp-raw Enabled Clone | Delete
issue:
1. only source type log4j worked with jetty jvm but whitelist not work
2. for main app jvm, it does not work even with log4j source type
3. jvm logging does not indicate anything about the agent (agent seems working since from splunk ui I did see the log4j data input from jetty jvm)
4. in Data Summary, when using "splunk_java_agent" or "tcp-raw" as source type, the splunk ui has never listed any of them as sourcetypes
5. in Data Summary, tcp:5250 was never listed as a source
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

On the Splunk side you need to open a TCP input for whatever port you have configured the agent to send to.
http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Monitornetworkports
