I started down this path too because I wanted to know if my indexers were behaving correctly as a daily health check.
First, I created a lookup from the results of "| tstats count where index=* by index,date_month,date_mday,date_wday,date_hour"
that looks back 13 weeks to collect the counts of events every hour in the past 13 weeks for every index. This runs on Sunday
morning. (I could have added index=_* too, but I haven't.)
Then, I created a lookup that calculates the average and standard deviation for each index for each day of the week that also
runs on Sunday morning after the first one, using it's data.
Finally, I joined a search of yesterday's results to the second lookup and can report on the indexes that aren't acting
normally by comparing yesterday's hourly counts using the average and standard deviation. I chose to use a line chart with
two lines (one for the averages and one for yesterday's counts). I created panels on a dashboard with charts for today (so far), yesterday, and this week -vs- average, as well as indexers that have deviated from norms and shown 0 events yesterday.
I may be able to post my dashboard after I've made sure it's working well, if my company allows it.
... View more