Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performance

[CommunityUser]

Hi all,

I tried to monitor hbase on Appdynamics using machine agent (App Server is already running in it) as per the following documentations,

http://community.appdynamics.com/t5/eXchange-Community-AppDynamics/HBase-Monitoring-Extension/idi-p/...

https://docs.appdynamics.com/display/PRO40/Install+the+Standalone+Machine+Agent

As of logs are concerned everything is working fine as expected,

Using Java Version [1.7.0_75] for Agent
Using Agent Version [Machine Agent v4.0.6.0 GA Build Date 2015-05-11 20:56:44]
[Machine Agent] Resolved install dir location :/home/test/AppdynamicsMachineAgent
Machine Agent Install Directory :/home/test/AppdynamicsMachineAgent
Machine Agent Temp Directory :/home/test/AppdynamicsMachineAgent/tmp
Tasks Root Directory :/home/test/AppdynamicsMachineAgent/controlchannel
Logger Configuration :/home/test/AppdynamicsMachineAgent/conf/logging/log4j.xml
Redirecting all logging statements to the configured logger
[Machine Agent] Resolved install dir location :/home/test/AppdynamicsMachineAgent
Started AppDynamics Machine Agent Successfully.

Even In Controller UI,

Am able to see the TIER with an APP_SERVER and MACHINE_SERVER agents running with 100 %

But when I look under `Application Infrastructure Performance`, Am not able to see the custom metrics under the `Application Infrastructure Performance`.
(i.e) As per the documentation, Am not able to find the following,

**In the AppDynamics Metric Browser, look for: Application Infrastructure Performance | <Tier> | Custom Metrics | HBase | Status | <DB name>**

its the end log available in machine-agent.log under log folder.

[Agent-Scheduler-1] 16 May 2015 19:39:14,448 INFO SystemAgent - Set up agent re-registration task
[Agent-Scheduler-1] 16 May 2015 19:39:14,454 INFO SystemAgentConfigManager - Scheduling configuration refresh at an interval of 60 seconds
[Agent-Scheduler-1] 16 May 2015 19:39:14,455 INFO SystemAgentConfigManager - Configuration refresh task interval is 60 seconds
[Agent-Scheduler-1] 16 May 2015 19:39:14,455 INFO SystemAgent - Configuration manager successfully configured
[Agent-Scheduler-1] 16 May 2015 19:39:14,464 INFO SystemAgent - Started AppDynamics Machine Agent Successfully.
[Agent-Monitor-Scheduler-3] 16 May 2015 19:39:19,436 INFO PeriodicTaskRunner - Periodic Task - setup metric feed for [HBase Monitor Run Task]

What is the change I need to make to get the hbase monitoring???

Thanks and Regards,
Harry.

[Arun_Dasetty]

Hi Harry, To assist you further, Can you send the archive of logs folder at path /home/test/AppdynamicsMachineAgent/logs and also the monitorxml at path /home/test/AppdynamicsMachineAgent/monitors/HBaseMonitor/ Regards, Arun

[CommunityUser]

Thanks Arun,

PFA of the logs,

monitor.xml and screen shot is available in the following GITHUB link,

https://github.com/Appdynamics/hadoop-monitoring-extension/issues/1

We planned to recommed this tool to our management, So please get back to us ASAP.

Thanks and Regards,

Harry

[Arun_Dasetty]

Hi Harry,

Please check the behavior post restarting machine agent jvm with high heap settings:

cmd> java -Xms128m -Xmx256m -jar machineagent.jar

We understood that HBase server is in same machine where machine agent is running, if yes, please do modify the xml file to localhost instead ip in monitor.xml, also we assume that the server HBase is not running out of reosurces as we below error in logs:

[pool-538-thread-1] 17 May 2015 04:36:19,485 ERROR HBaseMonitor - Failed to connect to 192.168.62.57:10101
java.rmi.ServerError: Error occurred in server thread; nested exception is:

Caused by: java.lang.OutOfMemoryError: unable to create new native thread

[AD Thread Pool-Global0] 17 May 2015 10:29:53,093 ERROR SystemAgentPollingForUpdate - Fatal transport error while connecting to URL [/controller/instance/4/systemagentpolling]: java.net.SocketTimeoutException: Read timed out

[CommunityUser]

Sorry for this Out of Memory error description, It happens very recently, Before that It shows like 100% UP in Server and Machine agent but still there was no CUSTOM METRICS under TIER...

OUT OF MEMORY just happens today... There is no relation to this. 

LOOK AT THE IMAGE in GITHUB

[Arun_Dasetty]

Hi Harry,

We agree with you, we would like to check the behavior with change suggested, please run the machine agent with more heap memory and also send the machine agent logs in debug mode if issue still persists post restart of machine agent and HBase server

[CommunityUser]

Arun / Radha,

We increased the memory in node and restartd the controller, AppServer agent, Machine Agent (Where HBASE_MONITOR_EXTENSION) is running. 

In Servers -> Tier (cvp) -> it shows as 100 % in Machine agent, App Server agent.

But as per the image, Custom metrics is not listed under Application Infrastructure Performance -> CVP -> 

CUSTOM METRICS not available.

PFA of the machine_Agent_log.

Please check my monitor.xml in github. Please tell me why custom metrics for Hbase is not listing in it.

Thanks and Regards,

Harry.

[Arun_Dasetty]

Hi Harry,

Can you send the machine agent logs folder archive in debug mode with run time for 15 minutes, info level logs does not have any errors in recent restarts of logs

Change logging level value from info to debug in log4j.xml file at path /home/test/AppdynamicsMachineAgent/conf/logging/ 

[CommunityUser]

Hi Arun

PFA of the machine log in debug mode...