topic alert when one transaction duration more than average duration*10? in Splunk Search

alert when one transaction duration more than average duration*10?

sonicant — Thu, 19 Jul 2012 15:07:50 GMT

There are continuous transactions' log into Splunk.
Is it possible to let Splunk alert when some transaction's duration is more than 10-times the average duration?

For example, average duration is A, and if some transaction's duration is over 10A, then Splunk raises an alert.

I tried " | stats avg(duration) as avgdur | eval threshold=avgdur*10 | search duration > threshold" but it doesn't work because events became stats.

Any one can give some advice? Thanks!

Re: alert when one transaction duration more than average duration*10?

sideview — Thu, 19 Jul 2012 18:24:38 GMT

stats will transform the event rows themselves. However eventstats will leave the event rows alone, and it will add the stats rows onto every event.

<some search=""> | eventstats avg(duration) as avgdur | search duration > avgdur*10

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/eventstats

while on the topic you might look at streamstats which is similar but where eventstats will run statistics over the entire resultset, streamstats will for each row only compute the stats up to that row.

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/streamstats

Re: alert when one transaction duration more than average duration*10?

richcollier — Wed, 27 Feb 2013 20:03:54 GMT

Is there a typo in your answer? Should it really be:

| eventstats avg(duration) as avgdur | search duration > avgdur*10

Re: alert when one transaction duration more than average duration*10?

sideview — Wed, 27 Feb 2013 20:23:37 GMT

You're absolutely right. Thanks. I'll fix the typo in the answer.

Re: alert when one transaction duration more than average duration*10?

richcollier — Wed, 27 Feb 2013 22:30:15 GMT

I also couldn't get your example (using eventstats) to work on real data where I knew I had a spike that was about 2.5x larger than the average...

However, I could get it to work using streamstats:

| streamstats avg(duration) as avgdur | where duration > avgdur*2.5

(had to use "where", not "search", by the way)

Re: alert when one transaction duration more than average duration*10?

richcollier — Wed, 27 Feb 2013 22:37:38 GMT

Of course, the real trick here is to be confident that 2.5 (or whatever the number you choose) is the right one. Set it too high and you'll miss potentially interesting spikes. Set it too low and you get bombarded.

Re: alert when one transaction duration more than average duration*10?

sideview — Wed, 27 Feb 2013 22:45:59 GMT

The two searches will behave very differently! streamstats will only calculate the average from the earliest point in the set up to the current row. And then each row might end up with a slightly different average. eventstats on the other hand will compute the average over the entire set, and then paint that single average onto every row.