How to calculate a per-second event count for a rolling time window?

[hexx]

I would like to display a per-second event count for a rolling time window, say 5 minutes.

I have tried the following approaches but without success :

• Using stats during a 5-minute window real-time search :

sourcetype=my_events | stats count as ecount | stats values(eval(ecount/300)) AS eps

=> This takes 5 minutes to give an accurate result. Until the search has run for the full length of the real-time window and "filled it" with events, the resulting EPS is inaccurate.

• Using stats and timechart with a span of 5 minutes during a 5-minute window real-time search :

sourcetype=my_events | bin _time span=5min | stats count | timechart span=5min per_second(count) AS eps

=> This discretizes results in 5 minute buckets, which is not what I want.

[hexx · Accepted solution]

This is currently a bit tricky. The first method mentioned (a simple stats dividing the event count by the search time window) is the one that should work but as of Splunk 4.2.2, real-time search windows do not back-fill with historical events that would match the window when the search is fired. This will however be possible in 4.2.3 and beyond.

In the meantime, you can achieve the desired result with the following search :

index=my_events | eval rt_window=300 | eval search_time=now() | eval seconds_elapsed=(_time - search_time) | eval secs=case(seconds_elapsed<0,"1",seconds_elapsed<rt_window,seconds_elapsed,seconds_elapsed>rt_window OR seconds_elapsed=rt_window,rt_window) | stats count as ecount, last(secs) AS seconds| stats values(ecount) AS "event count", values(seconds) AS "real-time search window (last X seconds)", values(eval(ecount/seconds)) AS eps

The logic behind this search is that we should divide the event count (the ecount field in this search) by the number of seconds that the time window spans (here rt_window, which is 300 seconds in the case of our 5-minute RT window) unless the search has not run for a full time window cycle yet. In that case, we will use eval case() to set the value of the divisor to the span of time that the search has run for (seconds_elapsed = _time - search_time).

Fortunately, this will be much easier to do in 4.2.3 with the RT-window back-fill option!