topic Re: failed to attach java agent to the running VM in Splunk AppDynamics

failed to attach java agent to the running VM

CommunityUser — Fri, 20 Oct 2017 06:14:11 GMT

I'm following the document to attach java agent to running VM.

But i get java exception while doing it, below is the output for my command:

[root@tropo-102 ~]# ps -ef | grep java | grep -v grep| awk '{print $2}'
10592
[root@tropo-102 ~]#

[root@tropo-102 ~]# /opt/voxeo/prism/jre/bin/java -Xbootclasspath/a:/opt/voxeo/prism/jre/lib/tools.jar -jar /root/appdynamics/javaagent.jar 10592
Attaching to VM [10592]
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:754)
Caused by: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
   at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
   at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
   at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
   ... 5 more
Exception in thread "main" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
   at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:768)
Caused by: java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:754)
Caused by: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
   at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
   at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
   at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
   ... 5 more

Any one aware of this issue? my Operation system is centOS 6.7

jack

Re: failed to attach java agent to the running VM

CommunityUser — Fri, 20 Oct 2017 06:17:12 GMT

BTW, i have change the owern of the agent to my application user in order to elimiate permission problem

Re: failed to attach java agent to the running VM

Peter_Holditch — Fri, 20 Oct 2017 15:20:13 GMT

What JVM version are you using? The symptoms look similar to those described here.

Re: failed to attach java agent to the running VM

CommunityUser — Mon, 23 Oct 2017 02:23:37 GMT

Hi,

Thanks for your feedback. My JDK is 1.8.0. Looks like the workaround is not work in my issue.

[root@tropo-102 ~]# /opt/voxeo/prism/jre/bin/java -version

java version "1.8.0_101"

Java(TM) SE Runtime Environment (build 1.8.0_101-b13)

Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

[root@tropo-102 ~]# ps -ef | grep java

voxeo 10592 10573 8 Oct20 ? 06:08:12 /opt/voxeo/prism/jre/bin/java -Dwrapper.key=OuYamULl31LbphFW -Dwrapper.port=32000 -Dvoxeo.config-dir=/opt/voxeo/prism/conf -Dproduct-name=prism -Dcom.sun.management.jmxremote -Djava.endorsed.dirs=/opt/voxeo/prism/common/endorsed -Djava.library.path=/opt/voxeo/prism/lib -Dkernel.home=/opt/voxeo/prism -Dcom.voxeolabs.aes.key=/opt/voxeo/prism/conf/aeskey -Dwrapper.native_library=wrapper -Dwrapper.version=3.2.3 -Dlog4j.ignoreTCL=true -Dfile.encoding=utf-8 -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/opt/voxeo/prism/conf/logging.properties -Dorg.tanukisoftware.wrapper.WrapperSimpleApp.waitForStartMain=TRUE -Dsun.net.inetaddr.ttl=60 -Dsun.net.client.defaultReadTimeout=60000 -Dsun.net.client.defaultConnectTimeout=60000 -Djava.rmi.server.RMIClassLoaderSpi=com.micromethod.sipmethod.server.SIPMethodRMIClassLoaderSpi -Djava.rmi.server.useCodebaseOnly=false -Dlog4j.configurationFile=/opt/voxeo/prism/conf/log4j2.xml -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -DAsyncLogger.WaitStrategy=Sleep -DAsyncLogger.RingBufferSize=262144 -Dcom.micromethod.sipmethod.server.cluster.logmc=false -Dh2.bindAddress=127.0.0.1 -Dcom.voxeo.sipmethod.server.launcher.main.block=true -Dnet.sf.ehcache.enableShutdownHook=true -Ddns.defaultQueryTimeout=10000 -Djava.security.egd=file:/dev/./urandom -Dcom.sun.phobos.javascript.useInterpreter=true -Dcom.voxeolabs.moho.suppressEarlyMedia=true -Dcom.voxeolabs.moho.copyHeadersForContinueRouting=false -classpath /opt/voxeo/prism/lib/jars/bcpkix-jdk15on-149b13.jar:/opt/voxeo/prism/lib/jars/bcprov-ext-jdk15on-149b13.jar:/opt/voxeo/prism/lib/jars/common-api.jar:/opt/voxeo/prism/lib/jars/commons-codec-1.4.jar:/opt/voxeo/prism/lib/jars/disruptor-3.3.5.jar:/opt/voxeo/prism/lib/jars/dom4j-1.6.1.jar:/opt/voxeo/prism/lib/jars/kernel-launcher.jar:/opt/voxeo/prism/lib/jars/kernel-utils.jar:/opt/voxeo/prism/lib/jars/log4j-1.2-api-2.6.2.jar:/opt/voxeo/prism/lib/jars/log4j-api-2.6.2.jar:/opt/voxeo/prism/lib/jars/log4j-core-2.6.2.jar:/opt/voxeo/prism/lib/jars/protobuf-java-2.6.0.jar:/opt/voxeo/prism/lib/jars/server-launcher.jar:/opt/voxeo/prism/lib/jars/server-tools.jar:/opt/voxeo/prism/lib/jars/server-utils.jar:/opt/voxeo/prism/lib/jars/servlet-api.jar:/opt/voxeo/prism/lib/jars/tropo-lic.jar:/opt/voxeo/prism/lib/jars/utils-apache.jar:/opt/voxeo/prism/lib/jars/vlib-core-12.2.25.jar:/opt/voxeo/prism/lib/jars/voxeolabs-util.jar:/opt/voxeo/prism/lib/jars/vsm-lib.jar:/opt/voxeo/prism/lib/jars/wrapper.jar:/opt/voxeo/prism/lib/jars/xmppservlet-api.jar:/opt/voxeo/prism/jre/lib/tools.jar:/opt/voxeo/prism/conf -server -Xms2G -Xmx2G -XX:+UseG1GC -XX:NewRatio=3 -XX:SurvivorRatio=8 -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m -Xloggc:/tropo_logs/gclog.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:ErrorFile=/tropo_logs/java_error_pid%p.log -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tropo_logs/java_heap_pid%p.hprof -XX:+PrintAdaptiveSizePolicy -XX:+ParallelRefProcEnabled org.tanukisoftware.wrapper.WrapperSimpleApp com.micromethod.sipmethod.server.Launcher -start

[root@tropo-102 ~]# /opt/voxeo/prism/jre/bin/java -XX:+StartAttachListener -Xbootclasspath/a:/opt/voxeo/prism/jre/lib/tools.jar -jar /root/appdynamics/javaagent.jar 10592

Attaching to VM [10592]

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:754)

Caused by: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded

at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)

at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)

at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)

... 5 more

Exception in thread "main" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:768)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:754)

Caused by: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded

at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)

at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)

at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)

... 5 more

Re: failed to attach java agent to the running VM

Peter_Holditch — Tue, 24 Oct 2017 12:36:14 GMT

Looking at the above command line, it seems that the java process you are targetting for instrumentation is the Tanuki simple wrapper.

By default, this will wait 2 seconds and then exit, after starting the actual application program which is is wrapping, so I wonder if it has already exitted by the time you run the attach?

I also wonder whether instrumenting the wrapper with AppDynamics is what you want to do? You are probably better instrumenting the process that the wrapper starts (or amending the wrapper.app.n properties to add in the -javaagent option to start the wrapped app together with the agent as it boots)