alert when one transaction duration more than average duration*10?

[sonicant]

There are continuous transactions' log into Splunk.
Is it possible to let Splunk alert when some transaction's duration is more than 10-times the average duration?

For example, average duration is A, and if some transaction's duration is over 10A, then Splunk raises an alert.

I tried " | stats avg(duration) as avgdur | eval threshold=avgdur*10 | search duration > threshold" but it doesn't work because events became stats.

Any one can give some advice? Thanks!

[sideview]

stats will transform the event rows themselves. However eventstats will leave the event rows alone, and it will add the stats rows onto every event.

<some search=""> | eventstats avg(duration) as avgdur | search duration > avgdur*10

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/eventstats

while on the topic you might look at streamstats which is similar but where eventstats will run statistics over the entire resultset, streamstats will for each row only compute the stats up to that row.

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/streamstats

[sideview]

The two searches will behave very differently! streamstats will only calculate the average from the earliest point in the set up to the current row. And then each row might end up with a slightly different average. eventstats on the other hand will compute the average over the entire set, and then paint that single average onto every row.

[richcollier]

Of course, the real trick here is to be confident that 2.5 (or whatever the number you choose) is the right one. Set it too high and you'll miss potentially interesting spikes. Set it too low and you get bombarded.

[richcollier]

I also couldn't get your example (using eventstats) to work on real data where I knew I had a spike that was about 2.5x larger than the average...

However, I could get it to work using streamstats:

| streamstats avg(duration) as avgdur | where duration > avgdur*2.5

(had to use "where", not "search", by the way)

[sideview]

You're absolutely right. Thanks. I'll fix the typo in the answer.

[richcollier]

Is there a typo in your answer? Should it really be:

| eventstats avg(duration) as avgdur | search duration > avgdur*10

??