I have a group of users to monitor. They create actions on a fairly regular basis, but they do not all follow the same pattern. Some perform this particular action 4x/hour, some 2x/hour, and some only 2x/day. What I would like to do is create a search that allows me to compare the previous hour of activity to the last 30 days and determine whether the past hour of activity is within "normal" or if their activity has dropped below a threshold.
So my thought is to calculate the avg and stdev for each hour of the day (0-23), per user. Then I could compare that data in the same search to the previous hour and see whether historical_avg_for_given_user_for_given_hour - historical_stdev_for_given_user_for_given_hour > activity_count_for_given_user_for_last_hour and if so, raise an alert for that user.
So that's the goal. I have been looking at "Time After Time – Comparing Time Ranges in Splunk" from @Anonymous, but I'm not sure how to apply that to a multi-user scenario.
Here's what I have so far:
[base search] earliest=-14d@d latest=-1d@d
| eval time_hour = strftime(_time, "%H")
| eval time_day = strftime(_time, "%D")
| stats count AS count_perhour_perday_peruser BY time_hour, time_day, userName
| chart limit=0 avg(count_perhour_perday_peruser) BY time_hour, userName
... View more