JMX ActiveSession metric collection deactivated after Tomcat restart for few instances

Stephane_Babouh — Wed, 30 Nov 2022 13:47:41 GMT

Hello,

we have a Tiers application with 4 nodes corresponding to Tomcat applications. We restart these applications every evening at 22:00. For 2 of 4 nodes, metrics correspond to JMX.ActiveSession is no longer collected after startup until around 10:00 the day after. It is fixed without any action but comes again the day after.

4 Nodes are managed the same with chef deployment.

Here are versions of the components :

Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0
AppDynamics Controller build 22.10.1-611

Here is a screenshot extracted from Metrics Browser :

Thanks for your suggestion and analysis of the potential root-cause.

Re: JMX ActiveSession metric collection desactivated afetr Tomcat restart for few instances

Stephane_Babouh — Wed, 30 Nov 2022 13:48:58 GMT

Here are screenshot extracted from Metrics Browser :

Re: JMX ActiveSession metric collection desactivated afetr Tomcat restart for few instances

Stephane_Babouh — Wed, 30 Nov 2022 13:52:32 GMT

Here is part of agent log file with error raised :

[AD Thread-Metric Reporter1] 28 Nov 2022 22:21:27,700 WARN AgentErrorProcessor - Agent error occurred, [name,transformId]=[com.singularity.JMX.JMXDomainFinder - java.lang.ClassNotFoundException,2147483647]
[AD Thread-Metric Reporter1] 28 Nov 2022 22:21:27,700 WARN AgentErrorProcessor - 4 instance(s) remaining before error log is silenced
[AD Thread-Metric Reporter1] 28 Nov 2022 22:21:27,700 ERROR JMXDomainFinder - Error occurred while trying to query MBean Server during version check -
java.lang.ClassNotFoundException: com.ibm.ws.security.core.SecurityContext
at com.sun.jmx.mbeanserver.ClassLoaderRepositorySupport.loadClass(ClassLoaderRepositorySupport.java:232) ~[?:1.8.0_232]
at com.sun.jmx.mbeanserver.ClassLoaderRepositorySupport.loadClass(ClassLoaderRepositorySupport.java:144) ~[?:1.8.0_232]
at com.sun.jmx.mbeanserver.SecureClassLoaderRepository.loadClass(SecureClassLoaderRepository.java:52) ~[?:1.8.0_232]
at com.singularity.ee.agent.appagent.services.jmxservice.appserver.JMXDomainsFinder.tryLoadingClassFromMBeanServerClassLoaderRepository(JMXDomainsFinder.java:347) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at com.singularity.ee.agent.appagent.services.jmxservice.appserver.JMXDomainsFinder.getAttributeFromWebSphereMBeanServer(JMXDomainsFinder.java:328) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at com.singularity.ee.agent.appagent.services.jmxservice.appserver.JMXDomainsFinder.checkIfMBeanServerSupportsMBeanServerInfo(JMXDomainsFinder.java:254) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at com.singularity.ee.agent.appagent.services.jmxservice.appserver.JMXDomainsFinder.loadAndGetDomainMapping(JMXDomainsFinder.java:111) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at com.singularity.ee.agent.appagent.services.jmxservice.appserver.common.ServerMBeanManager.run(ServerMBeanManager.java:236) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_232]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRun(ADFutureTask.java:320) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.run(ADFutureTask.java:140) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$301(ADScheduledThreadPoolExecutor.java:119) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:238) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726) ~[appagent.jar:Server Agent #22.5.0.33845 v22.5.0 GA compatible with 4.4.1.0 rd9770530415f19f4c5154a80772b833db8dd7cee release/22.5.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

Re: JMX ActiveSession metric collection deactivated after Tomcat restart for few instances

iamryan — Thu, 01 Dec 2022 23:30:21 GMT

Hi @Stephane.Babouhot,

I found this info. I hope it helps out or at leaves gives you a lead.

The JMX metrics are published by the JVM and javaagent fetches are reports those metrics (without any computation).

Could you please:

has a custom JMX rule configured for the activeSession?
check mbean browser (https://docs.appdynamics.com/display/PRO45/Configure+JMX+Metrics+from+MBeans - access mbean browser) to see what is the value of active session being reported there
check with the application team as to what is the exact mbean pattern where the application/container/server is publishing the activeSession. It could be possible the actual activeSession count is being published under some other mbean. In that case, we will need to configure custom JMX rule on that mbean and attribute.

topic Re: JMX ActiveSession metric collection deactivated after Tomcat restart for few instances in Splunk AppDynamics

JMX ActiveSession metric collection deactivated after Tomcat restart for few instances

Re: JMX ActiveSession metric collection desactivated afetr Tomcat restart for few instances

Re: JMX ActiveSession metric collection desactivated afetr Tomcat restart for few instances

Re: JMX ActiveSession metric collection deactivated after Tomcat restart for few instances