Is there a way to have Splunk send email alerts based on historical trend data?

[jframone]

Hi - I have been looking around for a way to do this, I'm not sure if it even exists.

Basically, I'm looking to see if there is a way to have Splunk send email alerts based on historical trend data.

For example, my current search is the following:

index=ui "webui03" "production_in_one_line.log" "Services::UploadController" | timechart avg(duration) as avg| where avg >=2800

** Search above checks the log file and outputs the latency number on the "duration" field, when I only display avg duration greater or equal than 2800ms.

Is there a way to include a historical trend option to monitor the log history and check for abnormal latency spikes? The reason why I'm looking to accomplish this is to avoid defining a threshold when creating the email alert in Splunk.

Any advise? Thanks.

[sundareshr]

The short answer is yes, there is a way. Now, how to do it, depends on how you want to define your threshold What I mean is, you can (below are just some ideas to get you going... these may not all be working samples)

use Standard Deviation (http://docs.splunk.com/Documentation/Splunk/6.5.0/SearchReference/CommonStatsFunctions)

index=ui "webui03" "production_in_one_line.log" "Services::UploadController" earliest=@d | timechart span=1h avg(duration) as hourly_avg | stats latest(hourly_avg) as hourly_avg stdev(hourly_avg) as hourly_stdev | where hourly_avg>hourly_stdev*2

OR, you can use trendline (https://docs.splunk.com/Documentation/Splunk/6.5.0/SearchReference/Trendline)

index=ui "webui03" "production_in_one_line.log" "Services::UploadController"  earliest=@d | timechart span=1h avg(duration) as hourly_avg | trendline sma2(hourly_avg) as trend | stats latest(duration) as latest latest(trend) as trend | where latest>trend*1.5

OR, a simple mean over time

index=ui "webui03" "production_in_one_line.log" "Services::UploadController"  earliest=@d | timechart span=1h max(duration) as max_duration | eventstats avg(duration) as avg_duration | where max_duration>avg_duration*2