There are continuous transactions' log into Splunk.
Is it possible to let Splunk alert when some transaction's duration is more than 10-times the average duration?
For example, average duration is A, and if some transaction's duration is over 10A, then Splunk raises an alert.
I tried "
Any one can give some advice? Thanks!
stats will transform the event rows themselves. However
eventstats will leave the event rows alone, and it will add the stats rows onto every event.
<some search=""> | eventstats avg(duration) as avgdur | search duration > avgdur*10
while on the topic you might look at
streamstats which is similar but where
eventstats will run statistics over the entire resultset,
streamstats will for each row only compute the stats up to that row.
The two searches will behave very differently! streamstats will only calculate the average from the earliest point in the set up to the current row. And then each row might end up with a slightly different average. eventstats on the other hand will compute the average over the entire set, and then paint that single average onto every row.
Of course, the real trick here is to be confident that 2.5 (or whatever the number you choose) is the right one. Set it too high and you'll miss potentially interesting spikes. Set it too low and you get bombarded.
I also couldn't get your example (using eventstats) to work on real data where I knew I had a spike that was about 2.5x larger than the average...
However, I could get it to work using streamstats:
(had to use "where", not "search", by the way)