AppD Archive

Alerting on rate of change in JMX metrics

CommunityUser
Splunk Employee
Splunk Employee

I'm new to AppDynamics but I havn't seen anything about this specifically so I apologize if this is fairly common knowledge. 

We have several counters that are exposed via JMX that track the number of times certian events occur (starting at 0 at JVM startup and incrementing each time the even occurs). We want to set a health rule that will trigger when the rate of change of that mettric crosses a certian threshhold (if the event occurs more than 10 times per minute trigger a Warn). The metric does not reset to 0 until we restart the JVM. 

Is it possible to calculate the rate of change in AppDynamics or is this something will will need to calculate and expose from within our Java app?

0 Karma

Arun_Dasetty
Super Champion

Hi,

Feel free to ask queries, we will always be glad to help you. 

 

Are you looking for something like the rule referred in below screenshot? if not please provide screenshots and and example to better answer your query?

image.png

Regards,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

I don't think that is quite what I'm looking for but my understanding of Health Rules may be incorrect. 

The screenshot below shows my line of thought. 

The variable currentCount shows the value of the metric at the current moment. The variable previousCount contains the value of the metric from the last time the rule was evaluated (which from my understanding is every minute). 

We want to know if the rate of chage between the two values is greater than X/timeFrame (I think in this example would be 10 events per minute).

In this example we are looking at the number of times we have throttled a specific API, if the number of throttles over a specific period of time is too high it could mean that there is a problem in the system so we want to be warned (since we've tuned our throttles to a point where we don't think they will be engaged but we don't want to risk the system's bandwidth). 

The example for a successful warning/critical event would be something like this:

At minute 1 the the count of throttled events is 10, at minute 2 the count is 30. 30 (currentCount) minus 10 (previousCount) is a rate of chage of 20 events per minute. Since this is higher than our 10 per minute threshold we would want the warning to engage on the effected node. 

This may be a very crude way to accomplsih this but I'm not familiar with what AppDynamics is capable of yet. 

Thanks for your help!

Todd

image.jpeg

0 Karma

Arun_Dasetty
Super Champion

Hi Todd,

Are the currentCount and previousCount are available as metric and if yes, are you looking for an email alert Or want to issue some operation on agent jvm instance when currentCountMetricValue  - PreviousCountMetricValue > 10 , if yes check if below doc helps:

https://docs.appdynamics.com/display/PRO40/Troubleshoot+Health+Rule+Violations

https://docs.appdynamics.com/display/PRO40/Policies

https://docs.appdynamics.com/display/PRO40/Alert+and+Respond

https://docs.appdynamics.com/display/PRO40/Actions

If above said is not the case, did you mean you are looking for option in health rule that will tell difference of same metric same CallsPerMinute for particular BT as such in given period where the difference of Min and Max of same metric is higher than value 10 Or it is 10x ?

image.png

If you are looking for value of same metric 10 minutes back and latest value in recent 10 minutes than i am afraid to say we do not have direct option to define health rule for such case

Let us know if there is any disconnect in my understanding of the requirement here.

Regards,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee
It sounds like the solution we were hoping for is not available.
At present the only value exposed is the count at the current time. We were hoping to be able to compare the value that is currently exposed (current count) with the value of the same metric that occurred at some point in the past (previous count).
Since it sounds like the functionality to compare the value of the same metric over time is not directly available we are going to look at instrumenting this within the app.
Thanks!
Todd

0 Karma

Arun_Dasetty
Super Champion

Hi Todd,

 

Thanks for confirming the requiremnt, Yes you are right, there is no direct way as of now to compare two values of same metric using HR rule but this is what i could suggest as a workaround , see if that information helps:

 

a) create a custom metric that is result of difference of value for same metric (you can copy rest api link from metric browser and pass custom time refer doc below that retrieves data for same metric for different time periods and create custom metric that is result the difference of value metric and report as custom metric in UI

 

b) define Health rule on the custom metric which is already a different of same metric for time period say 1:01 PM and 1:10 PM data value and now you could be able define condition as per your requiment, see if that make sense to you

 

https://docs.appdynamics.com/display/PRO40/Use+the+AppDynamics+REST+API

https://docs.appdynamics.com/display/PRO40/Extensions+and+Custom+Metrics

 

Regards,

Arun

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...