topic Re: Java App agent 4.5.x not starting on tomcat 8.5 in Splunk AppDynamics

Java App agent 4.5.x not starting on tomcat 8.5

CommunityUser — Wed, 06 Mar 2019 07:27:52 GMT

Hi,

I am deploying/testing java app agent 4.5.x for the first time & finding it difficult to instrument a local tomcat 8.5 instance.

I got below error, when I noticed agent 4.5 uses log4j2 for logging

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to theconsole.

I used java argument to point to the log4j2 configuration for correct logging.

-Dlog4j.configurationFile="<agent_install_path>\ver4.5.7.25056\conf\logging\log4j2.xml"

But I received another set of errors & the agent never starts. Can anyone suggest why I see these errors & next actions to fix this?

2019-03-06 12:44:27,903 ERROR Error processing element ADRRAFAppender: CLASS_NOT_FOUND
2019-03-06 12:44:27,905 ERROR Error processing element ADRRAFAppender: CLASS_NOT_FOUND
2019-03-06 12:44:27,906 ERROR Error processing element ADRRAFAppender: CLASS_NOT_FOUND
2019-03-06 12:44:27,907 ERROR Error processing element ADRRAFAppender: CLASS_NOT_FOUND
2019-03-06 12:44:27,908 ERROR Error processing element ADRRAFAppender: CLASS_NOT_FOUND
2019-03-06 12:44:28,076 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
2019-03-06 12:44:28,221 ERROR Unable to locate appender DefaultAppender for logger
2019-03-06 12:44:28,223 ERROR Unable to locate appender DynamicServiceAppender for logger com.singularity.dynamicservice
2019-03-06 12:44:28,224 ERROR Unable to locate appender BCTAppender for logger com.singularity.BCTLogger
2019-03-06 12:44:28,225 ERROR Unable to locate appender DefaultAppender for logger com.singularity
2019-03-06 12:44:28,226 ERROR Unable to locate appender RESTAppender for logger com.singularity
2019-03-06 12:44:28,227 ERROR Unable to locate appender BusinessTransactionsLogger for logger com.singularity.BusinessTransactions
Full Agent Registration Info Resolver using selfService [false]
Full Agent Registration Info Resolver using selfService [false]
Full Agent Registration Info Resolver using application name [TomcatOracleSample1]
Full Agent Registration Info Resolver using tier name [TomcatSampleTier1]
Full Agent Registration Info Resolver using node name [TomcatSampleNode001]
Install Directory resolved to[C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0]
2019-03-06 12:44:30,251 ERROR Error processing element ADRRAFAppender: CLASS_NOT_FOUND
2019-03-06 12:44:30,252 ERROR Error processing element ADRRAFAppender: CLASS_NOT_FOUND
2019-03-06 12:44:30,253 ERROR Error processing element ADRRAFAppender: CLASS_NOT_FOUND
2019-03-06 12:44:30,255 ERROR Error processing element ADRRAFAppender: CLASS_NOT_FOUND
2019-03-06 12:44:30,256 ERROR Error processing element ADRRAFAppender: CLASS_NOT_FOUND
2019-03-06 12:44:30,379 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
2019-03-06 12:44:30,498 ERROR Unable to locate appender DefaultAppender for logger
2019-03-06 12:44:30,499 ERROR Unable to locate appender DynamicServiceAppender for logger com.singularity.dynamicservice
2019-03-06 12:44:30,500 ERROR Unable to locate appender BCTAppender for logger com.singularity.BCTLogger
2019-03-06 12:44:30,501 ERROR Unable to locate appender DefaultAppender for logger com.singularity
2019-03-06 12:44:30,502 ERROR Unable to locate appender RESTAppender for logger com.singularity
2019-03-06 12:44:30,503 ERROR Unable to locate appender BusinessTransactionsLogger for logger com.singularity.BusinessTransactions
2019-03-06 12:44:30,879 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
Agent conf directory set to [C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\ver4.5.7.25056\conf]
[Thread-0] Wed Mar 06 12:44:33 IST 2019[DEBUG]: AgentInstallManager - Full Agent Registration Info Resolver is running
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using selfService [false]
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using selfService [false]
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using application name [TomcatOracleSample1]
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using tier name [TomcatSampleTier1]
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using node name [TomcatSampleNode001]
[Thread-0] Wed Mar 06 12:44:34 IST 2019[DEBUG]: AgentInstallManager - Full Agent Registration Info Resolver finished running
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: AgentInstallManager - Agent runtime directory set to [C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\ver4.5.7.25056]
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: AgentInstallManager - Agent node directory set to [TomcatSampleNode001]
Agent runtime conf directory set to C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\ver4.5.7.25056\conf
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: AgentInstallManager - Agent runtime conf directory set to C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\ver4.5.7.25056\conf
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: JavaAgent - Using Java Agent Version [Server Agent #4.5.7.25056 v4.5.7 GA compatible with 4.4.1.0 r34248b8ae5a623b8fa307bef8be41b9854f0e2b9 pull-requests/1987]
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: JavaAgent - Running IBM Java Agent [No]
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: JavaAgent - Java Agent Directory [C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\ver4.5.7.25056]
[Thread-0] Wed Mar 06 12:44:34 IST 2019[INFO]: JavaAgent - Java Agent AppAgent directory [C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\ver4.5.7.25056]
Agent logging directory set to [C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\ver4.5.7.25056\logs\TomcatSampleNode001]
2019-03-06 12:44:34,235 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
2019-03-06 12:44:34,237 INFO Using file at C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\ver4.5.7.25056\conf\logging\log4j2.xml
2019-03-06 12:44:34,241 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
2019-03-06 12:44:36,731 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
Running obfuscated agent
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Server version: Apache Tomcat/8.5.38
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Server built: Feb 5 2019 11:42:42 UTC
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Server number: 8.5.38.0
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: OS Name: Windows 7
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: OS Version: 6.1
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Architecture: amd64
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Java Home: C:\Program Files\Java\jdk1.8.0_141\jre
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: JVM Version: 1.8.0_141-b32
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: JVM Vendor: Oracle Corporation
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: CATALINA_BASE: C:\Users\jainsai\Downloads\apache-tomcat-8.5.38
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: CATALINA_HOME: C:\Users\jainsai\Downloads\apache-tomcat-8.5.38
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dcatalina.base=C:\Users\jainsai\Downloads\apache-tomcat-8.5.38
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dcatalina.home=C:\Users\jainsai\Downloads\apache-tomcat-8.5.38
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dwtp.deploy=C:\Users\jainsai\Downloads\apache-tomcat-8.5.38\wtpwebapps
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Djava.endorsed.dirs=C:\Users\jainsai\Downloads\apache-tomcat-8.5.38\endorsed
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dlog4j.configurationFile=C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\ver4.5.7.25056\conf\logging\log4j2.xml
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -javaagent:C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\javaagent.jar
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dappdynamics.http.proxyHost=userproxy.intranet.db.com
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dappdynamics.http.proxyPort=8080
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dfile.encoding=Cp1252
Mar 06, 2019 12:45:31 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [C:\Program Files\Java\jdk1.8.0_141\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:/Program Files/Java/jdk1.8.0_141/bin/../jre/bin/server;C:/Program Files/Java/jdk1.8.0_141/bin/../jre/bin;C:/Program Files/Java/jdk1.8.0_141/bin/../jre/lib/amd64;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Program Files (x86)\Microsoft Application Virtualization Client;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Citrix\ICAService\;C:\Program Files (x86)\Citrix\System32\;C:\PROGRAM FILES\CITRIX\System32\;C:\PROGRAM FILES\CITRIX\ICAService\;C:\Program Files (x86)\Common Files\Citrix\System32;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\PROGRAM FILES\CITRIX\Virtual Desktop Agent\;C:\Program Files\Git\cmd;C:\Program Files\Git\bin\;C:\Program Files (x86)\Enterprise Vault\EVClient\;C:\Program Files\eclipse;C:\Program Files\Anaconda3;C:\Program Files\Anaconda3\Scripts;C:\Program Files\Anaconda3\Library\bin;C:\Users\Public\Documents\php\php-7.1;C:\Users\Public\Documents\oracle\instantclient_18_3\instantclient_18_3;C:\Users\Public\Documents\devtools;C:\Program Files\Java\jdk1.8.0_141\bin;C:\Program Files\eclipse;;.]
Mar 06, 2019 12:45:31 PM org.apache.catalina.startup.Catalina load
SEVERE: Catalina.start
org.apache.catalina.LifecycleException: Failed to initialize component [StandardServer[8005]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
at org.apache.catalina.startup.Catalina.load(Catalina.java:639)
at org.apache.catalina.startup.Catalina.load(Catalina.java:662)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
Caused by: javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:200)
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:152)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:232)
at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:120)
at org.apache.catalina.core.JreMemoryLeakPreventionListener.lifecycleEvent(JreMemoryLeakPreventionListener.java:426)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
... 8 more
Caused by: java.lang.ClassNotFoundException: org/apache/xerces/jaxp/DocumentBuilderFactoryImpl
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:124)
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:188)
... 15 more

Re: Java App agent 4.5.x not starting on tomcat 8.5

Peter_Holditch — Wed, 06 Mar 2019 11:47:42 GMT

Sailesh,

Can you post your entire command line please, many things look off in the output you pasted.

Did the agent start and work OK despite the original ERROR message?

Warm regards,

Peter

Re: Java App agent 4.5.x not starting on tomcat 8.5

CommunityUser — Thu, 07 Mar 2019 06:50:02 GMT

Hi Peter,

Here is the complete argument list -

-Dcatalina.base="C:\Users\jainsai\Downloads\apache-tomcat-8.5.38" -Dcatalina.home="C:\Users\jainsai\Downloads\apache-tomcat-8.5.38" -Dwtp.deploy="C:\Users\jainsai\Downloads\apache-tomcat-8.5.38\wtpwebapps" -Djava.endorsed.dirs="C:\Users\jainsai\Downloads\apache-tomcat-8.5.38\endorsed" -Dlog4j.configurationFile="C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\ver4.5.7.25056\conf\logging\log4j2.xml" -javaagent:"C:\Users\jainsai\Downloads\Appdynamics 4.5\agent_bundles\dbapm-oracle-java-app-agent-v4.0\javaagent.jar" -Dappdynamics.http.proxyHost="userproxy.intranet.db.com" -Dappdynamics.http.proxyPort="8080"

Agent didn't start at all. Also tomcat server failed to start.

Re: Java App agent 4.5.x not starting on tomcat 8.5

Peter_Holditch — Thu, 07 Mar 2019 14:58:55 GMT

Does the app server start fine without the extra -D option for log4j.configurationFile? (other than the error message, which can be ignored)

Warm regards,

Peter

Re: Java App agent 4.5.x not starting on tomcat 8.5

CommunityUser — Fri, 08 Mar 2019 03:57:48 GMT

No, the app server doesn't start without or with the extra -D option for log4j.configurationFile.

Withour the the extra -D option for log4j.configurationFile, it throws below error -

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to theconsole.

Seems it is a known bug. Have a look at this support ticket I raised - http://help.appdynamics.com/hc/requests/173393

Re: Java App agent 4.5.x not starting on tomcat 8.5

Peter_Holditch — Fri, 08 Mar 2019 12:02:50 GMT

Good, I'm glad this is root-caused.

Re: Java App agent 4.5.x not starting on tomcat 8.5

Christopher_Sav — Wed, 12 Jun 2019 16:19:01 GMT

What is the root cause?

https://help.appdynamics.com/hc/en-us/requests/173393 goes to a 404 page

Re: Java App agent 4.5.x not starting on tomcat 8.5

CommunityUser — Thu, 04 Jul 2019 14:33:14 GMT

Kindly let us know what is the root cause. I am getting permission denied for the URL. I am also facing the same issue.

Re: Java App agent 4.5.x not starting on tomcat 8.5

Peter_Holditch — Thu, 04 Jul 2019 15:18:22 GMT

There were issues with how a few of the early 4.5.x agents used log4j

This issue is fixed in any recent 4.5.x agent (the most recent is 4.5.11)

If you experience the same issue with v4.5.11 of the agent, please create a new post with details, or open a support ticket.

Warm regards,

Peter

Re: Java App agent 4.5.x not starting on tomcat 8.5

CommunityUser — Fri, 05 Jul 2019 04:40:43 GMT

Thanks for the reply Peter.

If I am not wrong, any of the 4.5.x uses version 2 of log4j (i.e. log4j2) for logging.

Is there any way I can change this behaviour & make the agents to use log4j logging?

Regards,

Sailesh

Re: Java App agent 4.5.x not starting on tomcat 8.5

Peter_Holditch — Fri, 05 Jul 2019 17:17:58 GMT

Sailesh,

You should not need to, since agent logging and application / container logging should be entirely separate. If you find that not to be the case, I would really appreciate it if you could file a ticket, or post full details of your scenario.

That said, you can switch the agent back to using log4j by setting the appdynamics.agent.log4j2.disabled system property to true, but you should not need to do this.

Warm regards,

Peter