Splunk Search

alert when one transaction duration more than average duration*10?

Path Finder

There are continuous transactions' log into Splunk.
Is it possible to let Splunk alert when some transaction's duration is more than 10-times the average duration?

For example, average duration is A, and if some transaction's duration is over 10A, then Splunk raises an alert.

I tried " | stats avg(duration) as avgdur | eval threshold=avgdur*10 | search duration > threshold" but it doesn't work because events became stats.

Any one can give some advice? Thanks!

Tags (3)
0 Karma

SplunkTrust
SplunkTrust

stats will transform the event rows themselves. However eventstats will leave the event rows alone, and it will add the stats rows onto every event.

<some search=""> | eventstats avg(duration) as avgdur | search duration > avgdur*10

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/eventstats

while on the topic you might look at streamstats which is similar but where eventstats will run statistics over the entire resultset, streamstats will for each row only compute the stats up to that row.

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/streamstats

SplunkTrust
SplunkTrust

The two searches will behave very differently! streamstats will only calculate the average from the earliest point in the set up to the current row. And then each row might end up with a slightly different average. eventstats on the other hand will compute the average over the entire set, and then paint that single average onto every row.

0 Karma

Path Finder

Of course, the real trick here is to be confident that 2.5 (or whatever the number you choose) is the right one. Set it too high and you'll miss potentially interesting spikes. Set it too low and you get bombarded.

0 Karma

Path Finder

I also couldn't get your example (using eventstats) to work on real data where I knew I had a spike that was about 2.5x larger than the average...

However, I could get it to work using streamstats:

| streamstats avg(duration) as avgdur | where duration > avgdur*2.5

(had to use "where", not "search", by the way)

0 Karma

SplunkTrust
SplunkTrust

You're absolutely right. Thanks. I'll fix the typo in the answer.

0 Karma

Path Finder

Is there a typo in your answer? Should it really be:

| eventstats avg(duration) as avgdur | search duration > avgdur*10

??

0 Karma