AppDynamics Lite alerts not working for Websphere PMI Mbeans

[CommunityUser]

Hi,
First of all, thanks a ton for such a wonderful product.Currently we are using Lite version and we have plans to move to PRO soon. It will be great if you can help with the issue we are facing. We are trying to configure alerts from AppDynamics Lite for JMX Mbeans (Websphere PMI stats) but we see that the alert that is set if not reflected properly under "Trending & Alerting". We see that '-' appears under 'Current' & 'Average'.

1. The jetty logs (jetty.log under the logs folder) in Lite viewer continuously shows the following.

Mar 5, 2014 7:37:42 PM com.appdynamics.lite.viewer.core.r a
WARNING: The rule has not been evaluated since the metric is not registered yet. WebSpherePMI:path0=threadPoolModule,path1=WebContainer.ActiveCount
Mar 5, 2014 7:38:42 PM com.appdynamics.lite.viewer.core.r a
WARNING: The rule has not been evaluated since the metric is not registered yet. WebSpherePMI:path0=threadPoolModule,path1=WebContainer.ActiveCount
Mar 5, 2014 7:39:42 PM com.appdynamics.lite.viewer.core.r a
WARNING: The rule has not been evaluated since the metric is not registered yet. WebSpherePMI:path0=threadPoolModule,path1=WebContainer.ActiveCount
Mar 5, 2014 7:40:42 PM com.appdynamics.lite.viewer.core.r a
WARNING: The rule has not been evaluated since the metric is not registered yet. WebSpherePMI:path0=threadPoolModule,path1=WebContainer.ActiveCount

2. From the MBean Browser I am able to see the current values of the Mbean (the same active count webcontainer thread pool.). I am also seeing the current data using "Start Live graph". However the same is not happenning when I configure alerts.

3. Is it possible to customize the email message (e.g jvm name being displayed in the email.)

[Arun_Dasetty]

Hi,

Thanks for the feedback.  Please find the response to your queries:

>>> Q1: Answer: In first place we could not find the jmx alert rule "WebSpherePMI:path0=threadPoolModule,path1=WebContainer.ActiveCount" in alerts.xml file though we could see in liteviewer.log as follows:

[btpool0-25] 05 Mar 2014 18:07:47,658  INFO [btpool0-25] LiteAlertManager - Creating alert: WebSpherePMI:path0=threadPoolModule,path1=WebContainer.ActiveCount

and hence it is not working, but we see  the reason for alert not getting update could be due to the following error:

=========================

Viewer Lite Shell:> java.io.IOException: Bad file descriptor

[btpool0-17] 03 Mar 2014 11:28:25,902  WARN [btpool0-17] ConfigurationHandler - Error handling app agent configuration request
org.mortbay.jetty.EofException
    at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:767)
..
Caused by: java.net.SocketException: Broken pipe

==================

Can you please restart the liteviwer and re-install afresh if the restart does not help , check if you are using jdk version listed at http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6593729 as we see we could be hitting jdk bug here.

>>>> Q2: Answer: please provide the screenshots depicting the issue if restart helps to resolve the socket error in jetty.log and the alert is stored fine in alerts.xml

>>>> Q3:Answer: please provide the screenshots depicting the text you are referring to in email alert message.

Regards,

Arun

[CommunityUser]

Hi Akshay,

I see that the jvm version we are using is the right version (I am using IBM jvm) and also this time, I did not see the jetty related exception (not sure if there any issue at that time) but still the alerts are not working. I enabled DEBUG on Lite Viewer log4j and collected more logs if that helps.

I also see that the alert is added to the alerts.xml but still I keep getting the below in the logs.

WARNING: The rule has not been evaluated since the metric is not registered yet. WebSpherePMI:path0=threadPoolModule,path1=WebContainer.ActiveCount    (this is present as a part of the previous logs I had attached).

It says metric is not registered yet, should I have to do a config to register it? Again, I have no issue seeing the Live graphs for Websphere PMI though. Attaching the detailed logs. Appreciate your help in this. 

Regarding the question about email, I was just wondering if there is an option to customize the email message body of the alert.

Thanks

Rajarajan Pudupatti Sundari Jeyakodi

[CommunityUser]

Hi Akshay, Just to add the problem is not specific to Websphere but JMX. I see that none of the alerts configured for any JMX bean works. Just adding this point if it helps.

[Arun_Dasetty]

Hi,

We could see now alert persisted fine from alerts.xml and liteviewer.log as listed below:

============

[main] 06 Mar 2014 18:48:08,128  INFO [main] LiteAlertManager - Creating alert: WebSpherePMI:path0=threadPoolModule,path1=WebContainer.ActiveCount
    <alert>
        <alert-type>JMX</alert-type>
        <alert-name>WebSpherePMI:path0=threadPoolModule,path1=WebContainer.ActiveCount</alert-name>

=============

Can you please send the screenshot of the metric with live graph in liteviwer mbean browser UI for the above jmx metric  and also the debug agent logs from lite agent (you can change logging level in log4j.xml under <liteagent_dir>/conf/logging/ and also sed the liteviwer.logs in debug mode for the same period where live graph shows data in UI?

We see no option to customize the email content for alerts triggered by LITE version of our product.

Regards,

Arun

[CommunityUser]

Hi Arun, Thanks for all your help. Attached are the debug logs & screen shots you had requested. The debugs logs of both agent and viewer are attached here.

Screen Shot1 (1) - This shows via Mbean browser I can see the actual values

Screen Shot2 (2) - This shows via Live Graph I can see the real time values.

Screen Shot3 (3) - This shows the alert graph showing '-' & '-' (and -1 when i point the arrow). You can also see that the default alerts (default ART alert) are working fine but they are not JMX based.

Screen Shot5 (5) - This shows that tx snapshots are being captured without any issue.

Kindly let me know if you need anything else that will help you out.

[Arun_Dasetty]

Hi,

We tried similar case (jmx custom metric created from mbean browser) and is working fine, can you reduce the silence period in alert details in edit mode of alert rule and also check if click "refresh" icon refreshes the data in alert graph

Regards,

Arun

[CommunityUser]

Hi Arun,
Please find the update on this problem. I am still facing the issue with WAS PMI.

1. I tried connecting to a tomcat instance (using IBM JDK) and I was able to set the alerts properly. The screen shot for this is "tomcat_sucessful.jpg"

I see the below logs in the agent logs.

[AD Thread Pool-Global1] 15 Mar 2014 16:25:14,519 INFO JMXMetricRepository - Added new JMX Rule [JMXMetricRule [ MBeanQuery [MBeanQuery [ domain [Catalina], mbeanPattern [Catalina:type=Manager,context=/docs,host=localhost], queryLogicalOperator [null], queryExpressions []]], metricCategory [lite], beanName [null], metricPath [null], instanceName [null], instanceIdentifier [null], name [maxInactiveInterval], domain [null] Attribute Definitions [ JMX Attribute Definition [mbeanAttributeName [maxInactiveInterval], metricName [maxInactiveInterval], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [REGULAR_COUNTER] ] ] ]]

I also see the below logs which I did not see when I tried with WAS PMI

[AD Thread Pool-Global0] 15 Mar 2014 16:22:01,872 INFO JMXService - JVM JMX Metric collection initialized with update interval [60] seconds
[AD Thread Pool-Global0] 15 Mar 2014 16:22:01,873 INFO ConfigurationChannel - Finished re-enabling agent services

[AD Thread Pool-Global0] 15 Mar 2014 16:23:57,074 INFO HotspotConfig - Hotspot snapshots disabled: controller version is not >= 3.3.4

[AD Thread Pool-Global0] 15 Mar 2014 16:24:17,553 INFO ServerMBeanManagerVersion2 - The following domains were discovered [Users, JMImplementation, com.sun.management, Catalina, java.lang, java.util.logging, ]
[AD Thread Pool-Global0] 15 Mar 2014 16:24:17,596 INFO ServerMBeanManagerVersion2 - Started JMX MBean Reporter

2. I again tried connecting to the WAS JVM, but this time I tried setting up alert for another JMX Bean that was exposed (not under Websphere PMI). This was under java.lang domain, Garbage Collector-->MarkSweepCompact . This is seen in the screen shot WAS_MarkSweepCompact.jpg

Setting alert for this Mbean works perfectly fine. This was a WAS JVM. I can see the below logs,

[AD Thread Pool-Global0] 15 Mar 2014 16:22:01,871 INFO JMXService - Server JMX metric collection initialized with update interval [15] seconds
[AD Thread Pool-Global0] 15 Mar 2014 16:22:01,871 WARN MemoryMetricGenerator - Current JVM GC MX Beans - MarkSweepCompact, not registered with JMX Service.P
lease add the appropriate minor collection bean name to JMX Service property key name : jmx-gc-minorCollectionBean
[AD Thread Pool-Global0] 15 Mar 2014 16:22:01,871 INFO MemoryMetricGenerator - Identified major collection bean :MarkSweepCompact
[AD Thread Pool-Global0] 15 Mar 2014 16:22:01,871 INFO MemoryMetricGenerator - Initialized Major Collection GC metrics
[AD Thread Pool-Global0] 15 Mar 2014 16:22:01,871 INFO MemoryMetricGenerator - Initialized Ovearll GC metrics
[AD Thread Pool-Global0] 15 Mar 2014 16:22:01,871 INFO MemoryMetricGenerator - Initialized Memory MX Beans [class storage, JIT code cache, JIT data cache, mi
scellaneous non-heap storage, Java heap]

Screenshot : WAS_MarkSweepCompact.jpg

3. However in the same WAS JVM, when I try setting the alert for a WAS PMI Mbean, I am facing the problem. Looks like this has something to do with querying the WAS PMI Mbeans.

I see the below in agent logs. Is the problem because of the mbeanPattern below?

[AD Thread Pool-Global0] 15 Mar 2014 16:38:55,393 INFO JMXMetricRepository - Added new JMX Rule [JMXMetricRule [ MBeanQuery [MBeanQuery [ domain [WebSpherePMI], mbeanPattern [WebSpherePMI:path0=connectionPoolModule,path1=Oracle JDBC Driver,path2=jdbc/dpaygatewayCoreDS], queryLogicalOperator [null], queryExpressions []]], metricCategory [lite], beanName [null], metricPath [null], instanceName [null], instanceIdentifier [null], name [PoolSize], domain [null] Attribute Definitions [ JMX Attribute Definition [mbeanAttributeName [PoolSize], metricName [PoolSize], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [REGULAR_COUNTER] ] ] ]]

SCREEN SHOT : WAS_PMI_Fail.jpg

Thanks
Rajarajan Pudupatti SJ

[Arun_Dasetty]

Hi Rajarajan,

We apologize for the inconvenience caused here, we could replicate similar issue in local using Lite version.

However we would like to inform that this is not an issue in our Pro version of product , we request you to give a try using free trial version of our pro product version and keep us posted if that helps here.

Regards,

Arun

[CommunityUser]

Hi Arun - I am happy you are able to reproduce the problem. I have already tried out the pro version and this problem is not coming there. We are planning to ask a quote for the pro vesion. However for the test environments, we are planning to use the Liteversion until we finalize on the pro version. Is it possible to get a fix or a workaround to make these alerts work?

[Arun_Dasetty]

Hi,

Thanks for the update, Unfortunately as this issue is not reproducible in pro version and there will not be lite with this fixes in future we request you to go ahead with Pro trial version. Hope that is fine with you.

Regards,

Arun