I have an index which has around 50,000 errors per day and I need to create an alert which will take the number of errors for the previous day, by the hour, and will compare it to the current day, by the hour, and the alert will fire if the current days hour is atleast 10% higher then the previous days hour.
Example:
Yesterday
7:00am-7:59am we had 10,000 errors
8:00am-8:59am we had 6,000 errors
9:00am-9:59am we had 4,000 errors
Today
7am we had 11,000 errors
8am we had 20,000 errors
9am we had 3,000 errors
The alert will fire for only 7am and 8am because the errors were 10% higher than the previous day.. I need this for all 24 hours
I have a previous thread here which is similar to what I'm looking for now, but this one is slightly different
http://answers.splunk.com/answers/302401/moving-average-of-errors-for-a-2-hour-time-period.html?minQuestionBodyLength=80
My current search
index=vertex7-access RTG_Error="500" | eval date_hour = strftime(_time, "%H") | where date_hour=0 | bucket _time span=1d | timechart count | where count > 1.1 * [ search index=vertex7-access RTG_Error="500" earliest=-1d@d latest=@d | eval date_hour = strftime(_time, "%H") | where date_hour=0 | bucket _time span=1d | stats count by _time | stats avg(count) as AvgDailyError500Count | return $AvgDailyError500Count ]
Lastly, should I create a separate alert for each hour?
... View more