java.lang.VerifyError on WebSphere v8.5.5 (jdk 1.7)

[CommunityUser]

Hi,

Trying out AppDynamics Lite (just downloaded version info is null) I do get connection to the WAS process and see some data in the viewer. But when i execute one of our custom servlets i get the following exception:

java.lang.VerifyError: JVMVRFY012 stack shape inconsistent; class=xxxServlet, method=pause(J)V, pc=0

Caused by: java.lang.VerifyError: JVMVRFY012 stack shape inconsistent; class="xxxServlet, method=pause(J)V, pc=0
    at java.lang.J9VMInternals.verifyImpl(Native Method)
    at java.lang.J9VMInternals.verify(J9VMInternals.java:93)
    at java.lang.J9VMInternals.initialize(J9VMInternals.java:170)
    at java.lang.J9VMInternals.newInstanceImpl(Native Method)
    at java.lang.Class.newInstance(Class.java:1600)
    at java.beans.Beans.instantiate(Beans.java:241)
    at java.beans.Beans.instantiate(Beans.java:89)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper$1.run(ServletWrapper.java:1483)
    at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadServlet(ServletWrapper.java:1472)
    ... 23 more

I Did follow the instructions as per: http://litedocs.appdynamics.com/display/ADLite/IBM+WebSphere

and added the java agent to the generic jvm args.

When I remove the java agent this error doesn't appear.

Why is this error coming up, did I miss something I should do ?

Environment:

WebSphere [IBMJAVA7 7.0.4.1 gm1318.03]Platform 8.5.5.0 [EXPRESS 8.5.5.0 gm1319.01] running with ...
Host Operating System is Windows Server 2008 R2, version 6.1
Java version = 1.7.0, Java Compiler = j9jit26, Java VM name = IBM J9 VM

Thanks

[Arun_Dasetty]

Hi Stefan,

We see that IBM JDK 1.7 is not officially supported by AppDynamics LITE version of our product and also IBM websphere server 8.x is not in supported list as per docs: http://litedocs.appdynamics.com/display/ADLite/Supported+Applications+and+Frameworks+for+Java

However our AD Pro version of app server agent supports IBM JDK 1.7 version.

But we such issues in the following cases:

1) I would say a sector containing the .jar file on the hard-drive is broken , Either

• due to file corruption on the disk I would say
• or corrupted download before installing

2) java.lang.VerifyError can be the result when you have compiled against a different library than you are using at runtime.

Can you please try the following and let us know if that helps:
- try adding the option "-Xverify:all"  in addition other jvm args to server this helps to get more information on error messages
- can you try seeting the jvm arg "--effort=4" in addition -javaagent parameter and restart the jvm and check how it goes

Please be sure that Lite does not support officialy this test env we need your support while we try few options , also please send the completed agent logs if the issue still persists.

Let us know if that information helps

Thanks,

Arun

[CommunityUser]

I think that neither of these options (1) and (2) are the matter here because it works perfectly fine if I leave out the AppDynamics javaagent arguments and start WebSphere with the same EAR deployed.

1st  attempt:

-javaagent:C:\Programs\AppDynamicsLite\AppServerAgentLite-ibm\javaagent.jar -XX:-UseSplitVerifier

Error 500: java.lang.Exception: java.lang.VerifyError: JVMVRFY012 stack shape inconsistent; class=xxxServlet, method=pause(J)V, pc=0

2nd attempt:

-XX:-UseSplitVerifier -Xverify:all -javaagent:C:\Programs\AppDynamicsLite\AppServerAgentLite-ibm\javaagent.jar --effort=4 Server won’t start: Failed with unknown –effort argument.

3rd attempt:

-XX:-UseSplitVerifier -Xverify:all -javaagent:C:\Programs\AppDynamicsLite\AppServerAgentLite-ibm\javaagent.jar

Server won’t start.

So no success with all of these options.

See attatched log files..

[Arun_Dasetty]

Hi Stefan,

Can you please try the following :

- create a another IBM Appserver profile with no custom -D jvm args or -XX jvm opts and try configuring lite agent using -javaagent entry and check how it goes, we requested this as we suspect some incorrect or conflicting jar loading in server classpath causing this issue.

Also let us know will it be possible for you to try once the server with AD Pro agent, you can get free trial at http://info.appdynamics.com/freetrial.html

Thanks,

Arun

[CommunityUser]

Well, creating a new AppServer profile and deploying an new helloworld like servlet it all worked.

The only thing in the VM args is now "-javaagent:C:\Programs\AppDynamicsLite\AppServerAgentLite-ibm\javaagent.jar"

So, next thing is to test it on a fully packed profile with all kinds of jdbc/jms/etc resources and a few EAR's deployed.

One thing I don't understand yet is when viewing request details in my test EAR (containing a servlet in a WAR project which calls a EJB in a EJB project) is that sometimes the EJB call is seen in the request details and sometimes not. The EJB is always called.

Ok, thanks so far, I'll continue evaluating..

[Arun_Dasetty]

Hi Stefan,

Glad to hear that configuring in new profile worked fine, we suspect some libs in old profile causing some semantic evaluation issues during compilation when running with agent, hence to narrow the issue we requested to try with only -javaagent in new profile that helped to confirm that the error raised is either due to some jvm arg or due to conflciting or compilation issue due to libs in old profile.

regarding query on servlet calling EJB , can you confirm whether the query is in context of agent or it is related to webapp, as we see such invocation should not cause any issue with agent.

Sure keep us posted if you see any issues further.

Thanks,

Arun