JVM Instrumentation Agent is a convenient monitoring tool for the community. But I could not find any document to configure splunk ui to retrieve the raw apm event streamed data in enterprise UI. The agent properties are well defined and documented, though.
DD, please also check following thread dump. It seems splunk agent blocked the thread (maybe in deadlock?):
"Processor-Thread-0" prio=5 tid=0x00007f8df9169000 nid=0x6307 waiting on condition [0x000000011d08a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000700186c78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:324)
at com.splunk.javaagent.SplunkJavaAgent.methodEntered(Unknown Source)
at com.xxx.Processor.getCookie(Processor.java)
at com.xxx.Processor.verify( Processor.java:216)
at com.xxx.Processor$1.run( Processor.java:79)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
DD, good catch!
Now I got VerifyException due to the agent which is causing the main app not booting up. I added -noverify jvm argument so booting up was fine but the method should be invoked was hanging. if I dont attach the agent, everything works fine. it seems there are incompatibility between jmx and the agent. any suggestion?
I don't understand why your are bringing up JMX now.
Is this a typo , did you mean JVM ?
If so , What Java runtime version are your using ? 6,7,8 ?
Your whitelist notation is wrong.
In my examples and the documentation I use "/" not "." as the package path seperator
Hi DD,
thanks for quick reply. I manually defined the sourcetype "splunk_java_agent" and "tcp_raw" as indicated but none of them works. Is there any extra step missed to configure "splunk_java_agent"? from your slides, I know this sourcetype should work.
here is the main app jvm splunkagent.properties. the main modifications from your default file is the host, port and whitelist
agent.app.name=verifier
agent.app.instance=verifierJVM1
splunk.transport.impl=com.splunk.javaagent.transport.SplunkTCPTransport
splunk.transport.tcp.host=172.16.3.1
splunk.transport.tcp.port=5250
splunk.transport.tcp.maxQueueSize=5MB
splunk.transport.tcp.dropEventsOnQueueFull=false
trace.whitelist=com.tealium.tiqverifier.selenium.UtagVerifyDataProcessor:verify,com.tealium.rest.endpoints.tiqverifier.VerifierConfigurationRestResource:runTestConfig
trace.blacklist=com/sun,sun/,java/,javax/,com/splunk/javaagent/
trace.methodEntered=true
trace.methodExited=true
trace.classLoaded=true
trace.errors=true
trace.hprof=true
trace.hprof.tempfile=mydump.hprof
trace.hprof.frequency=30
trace.jmx=true
trace.jmx.configfiles=jmx
trace.jmx.default.frequency=30
Hi DD, could you please help to identify my issue? here is what I did but not working:
the use case: two JVMs, one with Jetty and another with main application; both applied unique properties file with dedicated port # (e.g., 5150 for jetty agent and 5250 for main app) and whitelist from your provided
in Splunk Enterprise UI, I created two TCP configurations under "Local Inputs" as below. I could not figure out which source type I should use but copied your slides. the only worked source type is log4j (only for jetty jvm) while the whitelist does not work (it pull everything not only the classes/methods I specified in whitelist).
TCP port Host Restriction Source type Status Actions
5150 splunk_java_agent Enabled Clone | Delete
5250 tcp-raw Enabled Clone | Delete
issue:
1. only source type log4j worked with jetty jvm but whitelist not work
2. for main app jvm, it does not work even with log4j source type
3. jvm logging does not indicate anything about the agent (agent seems working since from splunk ui I did see the log4j data input from jetty jvm)
4. in Data Summary, when using "splunk_java_agent" or "tcp-raw" as source type, the splunk ui has never listed any of them as sourcetypes
5. in Data Summary, tcp:5250 was never listed as a source
On the Splunk side you need to open a TCP input for whatever port you have configured the agent to send to.
http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Monitornetworkports