All Apps and Add-ons

how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?

mikessun
New Member

JVM Instrumentation Agent is a convenient monitoring tool for the community. But I could not find any document to configure splunk ui to retrieve the raw apm event streamed data in enterprise UI. The agent properties are well defined and documented, though.

0 Karma

mikessun
New Member

DD, please also check following thread dump. It seems splunk agent blocked the thread (maybe in deadlock?):

"Processor-Thread-0" prio=5 tid=0x00007f8df9169000 nid=0x6307 waiting on condition [0x000000011d08a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000700186c78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:324)
at com.splunk.javaagent.SplunkJavaAgent.methodEntered(Unknown Source)
at com.xxx.Processor.getCookie(Processor.java)
at com.xxx.Processor.verify( Processor.java:216)
at com.xxx.Processor$1.run( Processor.java:79)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

0 Karma

mikessun
New Member

DD, good catch!

Now I got VerifyException due to the agent which is causing the main app not booting up. I added -noverify jvm argument so booting up was fine but the method should be invoked was hanging. if I dont attach the agent, everything works fine. it seems there are incompatibility between jmx and the agent. any suggestion?

0 Karma

Damien_Dallimor
Ultra Champion

I don't understand why your are bringing up JMX now.
Is this a typo , did you mean JVM ?

If so , What Java runtime version are your using ? 6,7,8 ?

0 Karma

Damien_Dallimor
Ultra Champion

Your whitelist notation is wrong.
In my examples and the documentation I use "/" not "." as the package path seperator

0 Karma

mikessun
New Member

Hi DD,

thanks for quick reply. I manually defined the sourcetype "splunk_java_agent" and "tcp_raw" as indicated but none of them works. Is there any extra step missed to configure "splunk_java_agent"? from your slides, I know this sourcetype should work.

here is the main app jvm splunkagent.properties. the main modifications from your default file is the host, port and whitelist


See the README for definition of the propertys

---------------------

Common Agent options

---------------------

agent.app.name=verifier
agent.app.instance=verifierJVM1

agent.userEventTags=key1=value1,key2=value2

-------------------------

Splunk Transport options

-------------------------

splunk.transport.internalQueueSize=10000

splunk.transport.impl=com.splunk.javaagent.transport.SplunkTCPTransport

splunk.transport.impl=com.splunk.javaagent.transport.SplunkStdOutTransport

splunk.transport.tcp.host=172.16.3.1
splunk.transport.tcp.port=5250
splunk.transport.tcp.maxQueueSize=5MB
splunk.transport.tcp.dropEventsOnQueueFull=false

-----------------------------------

Class/Method/Error Tracing options

-----------------------------------

trace.whitelist=com.tealium.tiqverifier.selenium.UtagVerifyDataProcessor:verify,com.tealium.rest.endpoints.tiqverifier.VerifierConfigurationRestResource:runTestConfig
trace.blacklist=com/sun,sun/,java/,javax/,com/splunk/javaagent/
trace.methodEntered=true
trace.methodExited=true
trace.classLoaded=true
trace.errors=true

------------------------------

HPROF Dump Collection options

------------------------------

periodically dump hprof file(using JMX operation call), read in file & parse, send events to Splunk, delete file

trace.hprof=true
trace.hprof.tempfile=mydump.hprof

trace.hprof.tempfile=/etc/tmp/dump.hprof

in seconds

trace.hprof.frequency=30

trace.hprof.recordtypes=2,3,4

by default , all hprof records will be traced, but you can provide a comma delimited list of

specific record types to trace, the numbers are decimal values from the hprof spec.

HPROF RECORD TYPE CODE

==================================

HPROF_UTF8 1

HPROF_LOAD_CLASS 2

HPROF_UNLOAD_CLASS 3

HPROF_FRAME 4

HPROF_TRACE 5

HPROF_ALLOC_SITES 6

HPROF_HEAP_SUMMARY 7

HPROF_START_THREAD 10

HPROF_END_THREAD 11

HPROF_HEAP_DUMP 12

HPROF_HEAP_DUMP_SEGMENT 28

HPROF_HEAP_DUMP_END 44

HPROF_CPU_SAMPLES 13

HPROF_CONTROL_SETTINGS 14

Heap dumps(codes 12 & 28) can have subrecords , so you can use ":" notation to reference these values

trace.hprof.recordtypes=12:1,12:32,12:33,12:34

HEAPDUMP SUB RECORD TYPE CODE

=========================================

SUBRECORD_GC_ROOT_UNKNOWN -1

SUBRECORD_GC_ROOT_JNI_GLOBAL 1

SUBRECORD_GC_ROOT_JNI_LOCAL 2

SUBRECORD_GC_ROOT_JAVA_FRAME 3

SUBRECORD_GC_ROOT_NATIVE_STACK 4

SUBRECORD_GC_ROOT_STICKY_CLASS 5

SUBRECORD_GC_ROOT_THREAD_BLOCK 6

SUBRECORD_GC_ROOT_MONITOR_USED 7

SUBRECORD_GC_ROOT_THREAD_OBJ 8

SUBRECORD_GC_CLASS_DUMP 32

SUBRECORD_GC_INSTANCE_DUMP 33

SUBRECORD_GC_OBJ_ARRAY_DUMP 34

SUBRECORD_GC_PRIM_ARRAY_DUMP 35

--------------------------------------------------------

JMX attribute/operation/notification collection options

--------------------------------------------------------

Embedded JMX polling , all other JMX config is in the JMX XML configuration file.

trace.jmx=true

name of XML files(minus the ".xml" suffix) that should reside in the root of splunkagent.jar

trace.jmx.configfiles=jmx

trace.jmx.configfiles=goo,foo

in seconds

trace.jmx.default.frequency=30

trace.jmx.goo.frequency=30

0 Karma

Damien_Dallimor
Ultra Champion

You should not use the log4j sourcetype.

You should define your own sourcetype when you setup the TCP input.

alt text

Can you post your splunkagent.properties files.

Perhaps you have an error.

0 Karma

mikessun
New Member

Hi DD, could you please help to identify my issue? here is what I did but not working:

the use case: two JVMs, one with Jetty and another with main application; both applied unique properties file with dedicated port # (e.g., 5150 for jetty agent and 5250 for main app) and whitelist from your provided

in Splunk Enterprise UI, I created two TCP configurations under "Local Inputs" as below. I could not figure out which source type I should use but copied your slides. the only worked source type is log4j (only for jetty jvm) while the whitelist does not work (it pull everything not only the classes/methods I specified in whitelist).

TCP port Host Restriction Source type Status Actions
5150 splunk_java_agent Enabled Clone | Delete
5250 tcp-raw Enabled Clone | Delete

issue:
1. only source type log4j worked with jetty jvm but whitelist not work
2. for main app jvm, it does not work even with log4j source type
3. jvm logging does not indicate anything about the agent (agent seems working since from splunk ui I did see the log4j data input from jetty jvm)
4. in Data Summary, when using "splunk_java_agent" or "tcp-raw" as source type, the splunk ui has never listed any of them as sourcetypes
5. in Data Summary, tcp:5250 was never listed as a source

0 Karma

Damien_Dallimor
Ultra Champion

On the Splunk side you need to open a TCP input for whatever port you have configured the agent to send to.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Monitornetworkports

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!