OutOfMemoryError: PermGen space

[CommunityUser]

When I enable the javaagent.jar in my web application, the server starts normally, but fails the first time I issue a request to the app. 

I tried a few things already.  I increased the permgen space 10x, which didn't resolve the issue.  And I tried the suggestion located at this stackoverflow thread. 

Neither of those things resolved it.  

Note that when I don't have the AppDynamics agent enabled, there are no issues with the permgen space.  

Does anyone have a suggestion on how to figure out the root of this problem?  And even better would be how to fix it!

Relevant specs: 

   JBoss EAP 6.3.3

   Oracle 11g

   Java web serivce running Spring and Hibernate

It appears to be consistently connected with Hibernate.   Here is a snippet of the stack trace (I also attached a more complete trace file): 

Caused by: java.lang.OutOfMemoryError: PermGen space
at java.lang.Class.getDeclaredFields0(Native Method) [rt.jar:1.7.0_71]
at java.lang.Class.privateGetDeclaredFields(Class.java:2499) [rt.jar:1.7.0_71]
at java.lang.Class.getField0(Class.java:2804) [rt.jar:1.7.0_71]
at java.lang.Class.getField(Class.java:1582) [rt.jar:1.7.0_71]
at javassist.util.proxy.ProxyFactory.setField(ProxyFactory.java:517) [javassist-3.12.1.GA.jar:3.12.0.GA]
at javassist.util.proxy.ProxyFactory.createClass3(ProxyFactory.java:502) [javassist-3.12.1.GA.jar:3.12.0.GA]
at javassist.util.proxy.ProxyFactory.createClass2(ProxyFactory.java:486) [javassist-3.12.1.GA.jar:3.12.0.GA]
at javassist.util.proxy.ProxyFactory.createClass1(ProxyFactory.java:422) [javassist-3.12.1.GA.jar:3.12.0.GA]
at javassist.util.proxy.ProxyFactory.createClass(ProxyFactory.java:394) [javassist-3.12.1.GA.jar:3.12.0.GA]
at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.getProxyFactory(JavassistLazyInitializer.java:163) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.proxy.pojo.javassist.JavassistProxyFactory.postInstantiate(JavassistProxyFactory.java:66) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFactory(PojoEntityTuplizer.java:187) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:170) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:77) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_71]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_71]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_71]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_71]
at org.hibernate.tuple.entity.EntityTuplizerFactory.constructTuplizer(EntityTuplizerFactory.java:107) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.tuple.entity.EntityTuplizerFactory.constructDefaultTuplizer(EntityTuplizerFactory.java:135) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping.<init>(EntityEntityModeToTuplizerMapping.java:80) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:323) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.persister.entity.AbstractEntityPersister.<init>(AbstractEntityPersister.java:475) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.persister.entity.SingleTableEntityPersister.<init>(SingleTableEntityPersister.java:133) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.persister.PersisterFactory.createClassPersister(PersisterFactory.java:84) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:297) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1385) [hibernate-core-3.5.6-Final.jar:3.5.6-Final]
at org.hibernate.cfg.AnnotationConfiguration.buildSessionFactory(AnnotationConfiguration.java:954) [hibernate-annotations-3.5.6-Final.jar:3.5.6-Final]
at org.openimmunizationsoftware.dqa.manager.OrganizationManager.getSessionFactory(OrganizationManager.java:20) [ois-dqa-core-1.5-SNAPSHOT.jar:]
at gov.mi.mdch.xfer.custom.service.DQAMessageServiceImpl.submitMessage(DQAMessageServiceImpl.java:55) [classes:]
at gov.mi.mdch.xfer.custom.service.MessageController.submitMessageForProcessing(MessageController.java:39) [classes:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_71]

[Arun_Dasetty]

Hi,

We do not see AD agent class/package com.singularity references in the error stack trace and that confirms that agent looks not be causing the issue here, However, Can you provide the archive version of <AppServerAgent_dir>/logs folder for our initial review?

[CommunityUser]

Sure.  See attached. Thanks for taking a look!

[CommunityUser]

Are you sure your permgen size has indeed increaed? Once your application starts up with appdynamics agent try viewing the 'memory' tab to check on the 'Memory Pools' in AppDynamics Dashboard. See image attached.

To increase PermGen, try adding -XX:PermSize=256m -XX:MaxPermSize=512m to your Java command line that starts your Application Server.

[Arun_Dasetty]

Hi, 

we do not see permsize jvm heap settings in jvm args as per agent logs, we request you to set to as suggested by Subramanian in previous update and see how it goes.