topic Re: How to Set an Alert on a Moving Average in Alerting

How to Set an Alert on a Moving Average

skoelpin — Tue, 11 Aug 2015 19:49:16 GMT

I am tracking 500 errors on a daily basis. The average usually remains constant but sometimes it will increase more than 50%. If this happens I want to have Splunk send an alert

My current search

index=vertex7-access RTG_Error="500" earliest=-6d@d latest=@d | timechart count | timewrap d

So if the moving average deviates more then 50% over the average for the past 6 days, I want Splunk to alert me

Re: How to Set an Alert on a Moving Average

somesoni2 — Tue, 11 Aug 2015 21:55:41 GMT

For moving averages, If you the search today (Aug-11), you want an alert if the avg from Aug 5-Aug 10, is 50% higher than avg from Aug 4-Aug9 ?

Re: How to Set an Alert on a Moving Average

skoelpin — Tue, 11 Aug 2015 23:07:07 GMT

Yes very close. I want an alert set on a cron schedule to run every 6 hours (It was originally 1 day but now I want it to be 6 hours). So the alert will have an average from [Aug 5th - Aug 10th] and if at anytime that number goes 50% above the ~6 day average then an alert is sent out

Re: How to Set an Alert on a Moving Average

skoelpin — Tue, 11 Aug 2015 23:19:02 GMT

To add onto this with an example

From [Aug 5th - Aug 10th] the average number of errors for the day was 90,000

Say on August 11th the average is 140,000, we are now 50% more than what our average was from [Aug 5th - Aug 10th] and an alert is sent

Re: How to Set an Alert on a Moving Average

woodcock — Wed, 12 Aug 2015 13:36:10 GMT

Run this search every 6 hours for the last 24 hours:

index=vertex7-access RTG_Error="500" | stats count | where count >  1.5 * [ search index=vertex7-access RTG_Error="500" earliest=-6d@d latest=@d | bucket _time span=1d | stats count by _time | stats avg(count) as AvgDailyError500Count | return $AvgDailyError500Count ]

Re: How to Set an Alert on a Moving Average

skoelpin — Wed, 12 Aug 2015 13:48:29 GMT

It's only returning the count over the past 6 days, how should I test this?

Do you recommend I change this ..| where count > 0.5 so an alert triggers at 50% of the current average?

Would it be possible to create a barchart showing the count for each day (over the past 6 days) and have a trendline showing the 6 day average. Then I can have the alert go off if it goes 50% higher?

Re: How to Set an Alert on a Moving Average

diogofgm — Wed, 12 Aug 2015 13:55:49 GMT

Hey skoelpin
Try this:

index=vertex7-access RTG_Error="500" earliest=-6d@d latest=-1@d | timechart span=1d count AS totals | stats avg(totals) AS last_week_avg | appendcols [search index=vertex7-access RTG_Error="500" earliest=-1d@d latest=now | timechart span=1d count AS today_avg] | eval alert = if((today_avg>last_week_avg*1.5),"true","false")

Then you just need to choose alert if custom condition is met: alert=true

There might be another solution without subsearches but this should work.

Re: How to Set an Alert on a Moving Average

woodcock — Wed, 12 Aug 2015 14:00:46 GMT

That is a reasonable way to test it.

You can take out the guts and chart it like this:

 index=vertex7-access RTG_Error="500" earliest=-6d@d latest=@d | bucket _time span=1d | timechart count

If you need a more compact view, use a sparkline like this:

 index=vertex7-access RTG_Error="500" earliest=-6d@d latest=@d |  stats sparkline(count, 1d)

Have you tried looking at the predict command?

http://info.prelert.com/blog/anomaly-detective-vs-splunks-anomalies-command-what-is-the-difference

Re: How to Set an Alert on a Moving Average

skoelpin — Thu, 13 Aug 2015 18:36:22 GMT

I tested this and it worked perfectly. Thanks!