I wanted to catch burst of events reaching certain threshold in a short period time. I think splunk must have this function but no luck finding it. Could you please point me to the doc?
I imagine the search would look like:
sourcetype=foo condition1=xxx condition2=yyy |where event count within 5 seconds > 100
You could use the trendline command which will compute "moving average":
sourcetype=foo condition1=xxx condition2=yyy | timechart count span=1s | trendline sma5(count) as moving_count_events | eval burst=if(count> 2 * moving_count_events, 9999999, 0)
Then, you can filter with where command:
... | where burst=9999999
Thanks so I searched with your suggested query, it seems good but it returns a table. How can I modify it to make it return events themselves?
sourcetype=foo condition1=xxx condition2=yyy | timechart count span=1s | trendline sma5(count) as moving_count_events |where burst=9999999
You can try replacing timechart by streamstats :
sourcetype=foo condition1=xxx condition2=yyy | streamstats count timewindow=1s| trendline sma5(count) as movingcount_events |where burst=9999999
got it solved finally. The answer is transaction.
sourcetype=foo condition1=xxx condition2=yyy |transaction field1 field2 field3 maxspan=5s maxpause=1s |where eventcount>100
Then you will see the events themselves.