topic Re: Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performanc in AppD Archive

Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performance

CommunityUser — Sat, 16 May 2015 14:54:04 GMT

Hi all,

I tried to monitor hbase on Appdynamics using machine agent (App Server is already running in it) as per the following documentations,

http://community.appdynamics.com/t5/eXchange-Community-AppDynamics/HBase-Monitoring-Extension/idi-p/829

https://docs.appdynamics.com/display/PRO40/Install+the+Standalone+Machine+Agent

As of logs are concerned everything is working fine as expected,

Using Java Version [1.7.0_75] for Agent
Using Agent Version [Machine Agent v4.0.6.0 GA Build Date 2015-05-11 20:56:44]
[Machine Agent] Resolved install dir location :/home/test/AppdynamicsMachineAgent
Machine Agent Install Directory :/home/test/AppdynamicsMachineAgent
Machine Agent Temp Directory :/home/test/AppdynamicsMachineAgent/tmp
Tasks Root Directory :/home/test/AppdynamicsMachineAgent/controlchannel
Logger Configuration :/home/test/AppdynamicsMachineAgent/conf/logging/log4j.xml
Redirecting all logging statements to the configured logger
[Machine Agent] Resolved install dir location :/home/test/AppdynamicsMachineAgent
Started AppDynamics Machine Agent Successfully.

Even In Controller UI,

Am able to see the TIER with an APP_SERVER and MACHINE_SERVER agents running with 100 %

But when I look under `Application Infrastructure Performance`, Am not able to see the custom metrics under the `Application Infrastructure Performance`.
(i.e) As per the documentation, Am not able to find the following,

its the end log available in machine-agent.log under log folder.

[Agent-Scheduler-1] 16 May 2015 19:39:14,448 INFO SystemAgent - Set up agent re-registration task
[Agent-Scheduler-1] 16 May 2015 19:39:14,454 INFO SystemAgentConfigManager - Scheduling configuration refresh at an interval of 60 seconds
[Agent-Scheduler-1] 16 May 2015 19:39:14,455 INFO SystemAgentConfigManager - Configuration refresh task interval is 60 seconds
[Agent-Scheduler-1] 16 May 2015 19:39:14,455 INFO SystemAgent - Configuration manager successfully configured
[Agent-Scheduler-1] 16 May 2015 19:39:14,464 INFO SystemAgent - Started AppDynamics Machine Agent Successfully.
[Agent-Monitor-Scheduler-3] 16 May 2015 19:39:19,436 INFO PeriodicTaskRunner - Periodic Task - setup metric feed for [HBase Monitor Run Task]

What is the change I need to make to get the hbase monitoring???

Thanks and Regards,
Harry.

Re: Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performanc

Arun_Dasetty — Mon, 18 May 2015 10:58:40 GMT

Hi Harry, To assist you further, Can you send the archive of logs folder at path /home/test/AppdynamicsMachineAgent/logs and also the monitorxml at path /home/test/AppdynamicsMachineAgent/monitors/HBaseMonitor/ Regards, Arun

Re: Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performanc

CommunityUser — Mon, 18 May 2015 12:02:20 GMT

Thanks Arun,

PFA of the logs,

monitor.xml and screen shot is available in the following GITHUB link,

https://github.com/Appdynamics/hadoop-monitoring-extension/issues/1

We planned to recommed this tool to our management, So please get back to us ASAP.

Thanks and Regards,

Harry

Re: Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performanc

Arun_Dasetty — Mon, 18 May 2015 12:13:07 GMT

Hi Harry,

Please check the behavior post restarting machine agent jvm with high heap settings:

cmd> java -Xms128m -Xmx256m -jar machineagent.jar

We understood that HBase server is in same machine where machine agent is running, if yes, please do modify the xml file to localhost instead ip in monitor.xml, also we assume that the server HBase is not running out of reosurces as we below error in logs:

[pool-538-thread-1] 17 May 2015 04:36:19,485 ERROR HBaseMonitor - Failed to connect to 192.168.62.57:10101
java.rmi.ServerError: Error occurred in server thread; nested exception is:

Caused by: java.lang.OutOfMemoryError: unable to create new native thread

[AD Thread Pool-Global0] 17 May 2015 10:29:53,093 ERROR SystemAgentPollingForUpdate - Fatal transport error while connecting to URL [/controller/instance/4/systemagentpolling]: java.net.SocketTimeoutException: Read timed out

Re: Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performanc

CommunityUser — Mon, 18 May 2015 13:53:36 GMT

Sorry for this Out of Memory error description, It happens very recently, Before that It shows like 100% UP in Server and Machine agent but still there was no CUSTOM METRICS under TIER...

OUT OF MEMORY just happens today... There is no relation to this.

LOOK AT THE IMAGE in GITHUB

Re: Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performanc

Arun_Dasetty — Mon, 18 May 2015 16:02:19 GMT

Hi Harry,

We agree with you, we would like to check the behavior with change suggested, please run the machine agent with more heap memory and also send the machine agent logs in debug mode if issue still persists post restart of machine agent and HBase server

Re: Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performanc

CommunityUser — Tue, 19 May 2015 07:36:23 GMT

Arun / Radha,

We increased the memory in node and restartd the controller, AppServer agent, Machine Agent (Where HBASE_MONITOR_EXTENSION) is running.

In Servers -> Tier (cvp) -> it shows as 100 % in Machine agent, App Server agent.

But as per the image, Custom metrics is not listed under Application Infrastructure Performance -> CVP ->

CUSTOM METRICS not available.

PFA of the machine_Agent_log.

Please check my monitor.xml in github. Please tell me why custom metrics for Hbase is not listing in it.

Thanks and Regards,

Harry.

Re: Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performanc

Arun_Dasetty — Tue, 19 May 2015 13:22:47 GMT

Hi Harry,

Can you send the machine agent logs folder archive in debug mode with run time for 15 minutes, info level logs does not have any errors in recent restarts of logs

Change logging level value from info to debug in log4j.xml file at path /home/test/AppdynamicsMachineAgent/conf/logging/

Re: Appdynamics custom metrics - hbase is not displaying under Application Infrastructure Performanc

CommunityUser — Thu, 21 May 2015 06:31:39 GMT

Hi Arun

PFA of the machine log in debug mode...