Splunk AppDynamics

Retrieving historic metrics and applying complex calculation logic.

Tony_Ursich
New Member

Hello Team,

I presently have a single numeric value that is published to AppD every day, below is a sample of this extract:

  • name=Custom Metrics|GRA|Pipeline View|WinSec|Total Input, value=7366, aggregator=AVERAGE, time-rollup=AVERAGE, cluster-rollup=INDIVIDUAL

Over the course of a month we will have a curve with 30x data points, as depicted by the screenshot below.

Below are the requirements that we are trying to achieve, but I do not know where to start with the design and implementation, and would really appreciate some tips:

----------------------------------------------------------------------

Variance Warning Alert

The warning alert will be based on a comparison of todays metric versus the metric from the previous day or week, based on the below rules

Resource: Daily cadence report located below.

Rules:

Sat total is compared to prev Sat total
Sun total is compared to prev Sun total
Mon total is compared to prev Mon total
Tues total is compared to Mon (y'day) total
Wed total is compared to Tues (y'day) total
Thur total is compared to Wed (y'day) total
Fri total is compared to Thurs (y'day) total

Calculation (Day to day variation):

allowableWarningMarginValue = todays metric * 0.5
allowableCriticalMarginValue = todays metric * 0.8
ingestVariation = Abs(todays total - comparison day total)
If (ingestVariation >= allowableWarningMarginValue and ingestVariation < allowableCriticalMarginValue) then
  set heatmap value = AMBER
  generate a warning message
Else If (ingestVariation >= allowableCriticalMarginValue) then
  set heatmap value = RED
  generate a critical error message
Else
set heatmap value = GREEN
Endif


Example (Using Friday June 23 WinSec pipeline):
Todays Total (Fri June 23) = 7366
Comparison Total (Thurs June 22) = 300
allowableWarningMarginValue = 7366 * 0.5 = 3683
allowableCriticalMarginValue = 7366 * 0.8 = 5892
ingestVariation = abs(7366 - 300) = 7066

if ingestVariation (7066) > allowableCriticalMarginValue (5892)
  set heatmap value = RED
  generate a critical message
else if ingestVariation (7066) > allowableWarningMarginValue (3683)
  set heatmap value = AMBER
  generate a warning message
else
set heatmap value = GREEN
endif


AppD Result:

For a given day I would like to simply output a single heatmap colour. The below screenshot illustrates how this heatmap colour would look over successive days, but each daily heatmap colour is simply overridden each day.

image.png

Thank you in advance for any help you can provide.

Kind regards,

Tony.

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

Watch On Demand the Tech Talk on November 6 at 11AM PT, and empower your SOC to reach new heights! Duration: ...

Splunk Observability as Code: From Zero to Dashboard

For the details on what Self-Service Observability and Observability as Code is, we have some awesome content ...