All Apps and Add-ons
Highlighted

how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?

New Member

JVM Instrumentation Agent is a convenient monitoring tool for the community. But I could not find any document to configure splunk ui to retrieve the raw apm event streamed data in enterprise UI. The agent properties are well defined and documented, though.

0 Karma
Highlighted

Re: how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?

Ultra Champion

On the Splunk side you need to open a TCP input for whatever port you have configured the agent to send to.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Monitornetworkports

0 Karma
Highlighted

Re: how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?

New Member

Hi DD, could you please help to identify my issue? here is what I did but not working:

the use case: two JVMs, one with Jetty and another with main application; both applied unique properties file with dedicated port # (e.g., 5150 for jetty agent and 5250 for main app) and whitelist from your provided

in Splunk Enterprise UI, I created two TCP configurations under "Local Inputs" as below. I could not figure out which source type I should use but copied your slides. the only worked source type is log4j (only for jetty jvm) while the whitelist does not work (it pull everything not only the classes/methods I specified in whitelist).

TCP port Host Restriction Source type Status Actions
5150 splunkjavaagent Enabled Clone | Delete
5250 tcp-raw Enabled Clone | Delete

issue:
1. only source type log4j worked with jetty jvm but whitelist not work
2. for main app jvm, it does not work even with log4j source type
3. jvm logging does not indicate anything about the agent (agent seems working since from splunk ui I did see the log4j data input from jetty jvm)
4. in Data Summary, when using "splunkjavaagent" or "tcp-raw" as source type, the splunk ui has never listed any of them as sourcetypes
5. in Data Summary, tcp:5250 was never listed as a source

0 Karma
Highlighted

Re: how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?

Ultra Champion

You should not use the log4j sourcetype.

You should define your own sourcetype when you setup the TCP input.

alt text

Can you post your splunkagent.properties files.

Perhaps you have an error.

0 Karma
Highlighted

Re: how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?

New Member

Hi DD,

thanks for quick reply. I manually defined the sourcetype "splunkjavaagent" and "tcpraw" as indicated but none of them works. Is there any extra step missed to configure "splunkjava_agent"? from your slides, I know this sourcetype should work.

here is the main app jvm splunkagent.properties. the main modifications from your default file is the host, port and whitelist


See the README for definition of the propertys

---------------------

Common Agent options

---------------------

agent.app.name=verifier
agent.app.instance=verifierJVM1

agent.userEventTags=key1=value1,key2=value2

-------------------------

Splunk Transport options

-------------------------

splunk.transport.internalQueueSize=10000

splunk.transport.impl=com.splunk.javaagent.transport.SplunkTCPTransport

splunk.transport.impl=com.splunk.javaagent.transport.SplunkStdOutTransport

splunk.transport.tcp.host=172.16.3.1
splunk.transport.tcp.port=5250
splunk.transport.tcp.maxQueueSize=5MB
splunk.transport.tcp.dropEventsOnQueueFull=false

-----------------------------------

Class/Method/Error Tracing options

-----------------------------------

trace.whitelist=com.tealium.tiqverifier.selenium.UtagVerifyDataProcessor:verify,com.tealium.rest.endpoints.tiqverifier.VerifierConfigurationRestResource:runTestConfig
trace.blacklist=com/sun,sun/,java/,javax/,com/splunk/javaagent/
trace.methodEntered=true
trace.methodExited=true
trace.classLoaded=true
trace.errors=true

------------------------------

HPROF Dump Collection options

------------------------------

periodically dump hprof file(using JMX operation call), read in file & parse, send events to Splunk, delete file

trace.hprof=true
trace.hprof.tempfile=mydump.hprof

trace.hprof.tempfile=/etc/tmp/dump.hprof

in seconds

trace.hprof.frequency=30

trace.hprof.recordtypes=2,3,4

by default , all hprof records will be traced, but you can provide a comma delimited list of

specific record types to trace, the numbers are decimal values from the hprof spec.

HPROF RECORD TYPE CODE

==================================

HPROF_UTF8 1

HPROFLOADCLASS 2

HPROFUNLOADCLASS 3

HPROF_FRAME 4

HPROF_TRACE 5

HPROFALLOCSITES 6

HPROFHEAPSUMMARY 7

HPROFSTARTTHREAD 10

HPROFENDTHREAD 11

HPROFHEAPDUMP 12

HPROFHEAPDUMP_SEGMENT 28

HPROFHEAPDUMP_END 44

HPROFCPUSAMPLES 13

HPROFCONTROLSETTINGS 14

Heap dumps(codes 12 & 28) can have subrecords , so you can use ":" notation to reference these values

trace.hprof.recordtypes=12:1,12:32,12:33,12:34

HEAPDUMP SUB RECORD TYPE CODE

=========================================

SUBRECORDGCROOT_UNKNOWN -1

SUBRECORDGCROOTJNIGLOBAL 1

SUBRECORDGCROOTJNILOCAL 2

SUBRECORDGCROOTJAVAFRAME 3

SUBRECORDGCROOTNATIVESTACK 4

SUBRECORDGCROOTSTICKYCLASS 5

SUBRECORDGCROOTTHREADBLOCK 6

SUBRECORDGCROOTMONITORUSED 7

SUBRECORDGCROOTTHREADOBJ 8

SUBRECORDGCCLASS_DUMP 32

SUBRECORDGCINSTANCE_DUMP 33

SUBRECORDGCOBJARRAYDUMP 34

SUBRECORDGCPRIMARRAYDUMP 35

--------------------------------------------------------

JMX attribute/operation/notification collection options

--------------------------------------------------------

Embedded JMX polling , all other JMX config is in the JMX XML configuration file.

trace.jmx=true

name of XML files(minus the ".xml" suffix) that should reside in the root of splunkagent.jar

trace.jmx.configfiles=jmx

trace.jmx.configfiles=goo,foo

in seconds

trace.jmx.default.frequency=30

trace.jmx.goo.frequency=30

0 Karma
Highlighted

Re: how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?

Ultra Champion

Your whitelist notation is wrong.
In my examples and the documentation I use "/" not "." as the package path seperator

0 Karma
Highlighted

Re: how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?

New Member

DD, good catch!

Now I got VerifyException due to the agent which is causing the main app not booting up. I added -noverify jvm argument so booting up was fine but the method should be invoked was hanging. if I dont attach the agent, everything works fine. it seems there are incompatibility between jmx and the agent. any suggestion?

0 Karma
Highlighted

Re: how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?

Ultra Champion

I don't understand why your are bringing up JMX now.
Is this a typo , did you mean JVM ?

If so , What Java runtime version are your using ? 6,7,8 ?

0 Karma
Highlighted

Re: how to configure splunk enterprise ui to receive raw apm event stream data sent from JVM Instrumentation Agent?

New Member

DD, please also check following thread dump. It seems splunk agent blocked the thread (maybe in deadlock?):

"Processor-Thread-0" prio=5 tid=0x00007f8df9169000 nid=0x6307 waiting on condition [0x000000011d08a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000700186c78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:324)
at com.splunk.javaagent.SplunkJavaAgent.methodEntered(Unknown Source)
at com.xxx.Processor.getCookie(Processor.java)
at com.xxx.Processor.verify( Processor.java:216)
at com.xxx.Processor$1.run( Processor.java:79)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

0 Karma