Java Agent fails: NoClassDefFoundError org.apache.logging.log4j.message.ReusableMessageFactory

[Frederic_Hansli]

Hello,

In K8S, on a pod running a Spring Boot 3.x application (with OpenJDK 17) auto-instrumented by cluster-agent, the Java Agent fails on startup:

[AD Agent init] Wed Sep 27 22:27:38 PDT 2023[INFO]: JavaAgent - Java Agent Directory [/opt/appdynamics-java/ver22.9.0.34210]
[AD Agent init] Wed Sep 27 22:27:38 PDT 2023[INFO]: JavaAgent - Java Agent AppAgent directory [/opt/appdynamics-java/ver22.9.0.34210]
Agent logging directory set to [/opt/appdynamics-java/ver22.9.0.34210/logs]
[AD Agent init] Wed Sep 27 22:27:38 PDT 2023[INFO]: JavaAgent - Agent logging directory set to [/opt/appdynamics-java/ver22.9.0.34210/logs]

Could not start Java Agent, disabling the agent with exception java.lang.NoClassDefFoundError: Could not initialize class org.apache.logging.log4j.message.ReusableMessageFactory, Please check log files

In the pod, the jar file (log4j-api) containing the ReusableMessageFactory is there (part of the appdynamics java-agent):

sh-4.4$ pwd
/opt/appdynamics-java/ver22.9.0.34210/lib/tp
sh-4.4$ ls log4j*
log4j-api-2.17.1.1.9.cached.packages.txt log4j-core-2.17.1.1.9.cached.packages.txt log4j-jcl-2.17.1.cached.packages.txt
log4j-api-2.17.1.1.9.jar log4j-core-2.17.1.1.9.jar log4j-jcl-2.17.1.jar
log4j-api-2.17.1.1.9.jar.asc log4j-core-2.17.1.1.9.jar.asc log4j-jcl-2.17.1.jar.asc

From the POD manifest:
- name: JAVA_TOOL_OPTIONS
value: ' -Dappdynamics.agent.accountAccessKey=$(APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY)
-Dappdynamics.agent.reuse.nodeName=true -Dappdynamics.socket.collection.bci.enable=true
-Dappdynamics.agent.startup.log.level=debug -Dappdynamics.agent.reuse.nodeName.prefix=eric-tmo-des-ms-entitlements
-javaagent:/opt/appdynamics-java/javaagent.jar'

I tried with the latest java-agent (23.9) but same result. I don't seem to have the problem with SpringBoot 2.7 (which does include log4j-api as opposed to 3.x). It seems the classloader can't find the class from the java-agent distribution.) 

Has anyone encountered this ? 

Thank you.

[Frederic_Hansli · Accepted solution]

For reasons I don't understand, the problem is fixed when I increase the JVM option --XXMaxJavaStackTraceDepth above 10. Any value below 10 causes the NoClassDefFoundError error.

If anyone has an explaination, much appreciated (does it have anything to do with maximum-activity-trace-stack default value of 10 ?