If CPU is > 95 for more than 5 minutes < -- how do you write the syntax?

[Jarohnimo]

Hello all,

I know it's possible to find values via greater than, less than or equal to in your search queries. Is it possible to add more arguments to request that the value has to be that way for at least 5 minutes, if so then return the results. In the end i wish to have an alert be sent out if any CPU is above 95% for more than 5 minutes, send an alert. We don't want an alert if its just hitting the ceilling and coming back down, we want it if it's on fire (a real problem).

I was trying something like

index=blah counter=% Processor Time" Sourcetype="perfmon:processor" Value > 95 span=5m | timechart etc.etc.etc...

I believe it's possible but i just need help getting the syntax correct.

Thanks in advance.

[DalJeanis · Accepted solution]

Here's the problem - If you limit your search results to >95, then you don't know that every result in a given time period would have been > 95. You only know that the max for that time period was at least 95. Instead, you want the min for the time period that you are checking over. Adjust the name of the cpu/host field as needed.

index=blah counter="% Processor Time" Sourcetype="perfmon:processor" 
earliest=-5m latest=now
| stats min(Value) as minValue by host
| where minValue>95 

[Jarohnimo]

So instead of limiting my results I filter it with where.. nice!!

My end game is an alert,
So the way I assume it will work is if any host has their CPU above 95%,
In real time search
that -5 is t starting point and the now is present, so as long as the value remains for atleast 5 minutes Splunk will alert me at the 5 minute mark?

Thanks

[DalJeanis]

@jarahnimo - you got it. Basically, the time range limits to the last 5 minutes, and then the min(Value) ensures that the CPU usage has not fallen below 95. Any CPU that didn't report at all, won't be included, but then their usage wasn't above 95%, so that would be a different alert anyway.

Now, you might want to verify how long, on average, your system takes to index and post the scores. If it takes an average of 5 seconds, and a 95th percentile of 8 seconds, then you might want to use...

earliest=-309s latest=-9s