I had a situation where I wanted to know if the mstats p90(cpu) over 5 minutes of a host was above a certain value; but needed to extend it to 10 minutes for some hosts. I figured rather than make two searches I could use span=5m and search back 10 minutes:
(Search Window: -10m@m to @m)
| mstats p90(_value) AS p90A WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=5m
Except this was often producing 3 events per host, because unless I'm mistaken mstats span always aligns to UTC 0, so if I'm running the search on a minute not divisible by 5 (say every 3 minutes) I'll end up with 3 data points per host instead of 2.
So I thought, maybe using prestats + bin + stats will work; I can get 10 samples and use bin aligntime=earliest to force them to just 2 time bins. I think this works, a quick check says that the P90 values are the same up until 4 decimal places if the times are aligned:
(Search Window: @h-10m to @h)
| mstats p90(_value) AS p90A WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=5m
| join host, _time
[| mstats p90(_value) prestats=true WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=1m
| bin _time span=5m aligntime=earliest
| stats p90(_value) AS p90B BY host, _time
]
| where round(p90A,4) != round(p90B,4)
So this search should work for any two 5 minute intervals aligned to any minute of the day.
| mstats p90(_value) prestats=true WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=1m
| bin _time span=5m aligntime=earliest
| stats p90(_value) AS p90 BY host, _time
| where p90 > 80
| stats list(p90), count by host
| where count == 2 OR match(host, "prod")
I ended up not needing it when I realized my alert was already locked to every 5 minutes. Has anyone else tried doing this? Know a better way without creating two searches?
