Splunk Search

How do I use mstats and changing aligntime to set up this alert

JustinSC
Explorer

I had a situation where I wanted to know if the mstats p90(cpu) over 5 minutes of a host was above a certain value; but needed to extend it to 10 minutes for some hosts. I figured rather than make two searches I could use span=5m and search back 10 minutes:

(Search Window: -10m@m to @m)

| mstats p90(_value) AS p90A WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=5m

Except this was often producing 3 events per host, because unless I'm mistaken  mstats span always aligns to UTC 0, so if I'm running the search on a minute not divisible by 5 (say every 3 minutes) I'll end up with 3 data points per host instead of 2.

So I thought, maybe using prestats + bin + stats will work; I can get 10 samples and use bin aligntime=earliest to force them to just 2 time bins. I think this works, a quick check says that the P90 values are the same up until 4 decimal places if the times are aligned:

(Search Window: @h-10m to @h)

| mstats p90(_value) AS p90A WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=5m
| join host, _time 
    [| mstats p90(_value) prestats=true WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=1m 
    | bin _time span=5m aligntime=earliest
    | stats p90(_value) AS p90B BY host, _time 
        ] 
| where round(p90A,4) != round(p90B,4)

So this search should work for any two 5 minute intervals aligned to any minute of the day.

| mstats p90(_value) prestats=true WHERE metric_name="Processor.%_Processor_Time" AND instance="_Total" BY host span=1m 
| bin _time span=5m aligntime=earliest 
| stats p90(_value) AS p90 BY host, _time 
| where p90 > 80 
| stats list(p90), count by host 
| where count == 2 OR match(host, "prod")

I ended up not needing it when I realized my alert was already locked to every 5 minutes. Has anyone else tried doing this? Know a better way without creating two searches?

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

Watch On Demand the Tech Talk on November 6 at 11AM PT, and empower your SOC to reach new heights! Duration: ...

Splunk Observability as Code: From Zero to Dashboard

For the details on what Self-Service Observability and Observability as Code is, we have some awesome content ...