*search criteria* earliest=-2m queued_jobs<10
Set it as an alert over a rolling window of two minutes, and tell it to alert you if the number of results equals 0. If there's ever a two minute window during which the field is constantly 10 or greater, you'll be alerted.
If you want it to return if the average is greater than 10, it's a bit trickier. Try this:
*search criteria* earliest=-2m | outlier |stats avg(queued_jobs) as average
Like before, set it as an alert over a rolling window of two minutes. This time, set it to use a custom condition is met. Set the custom condition to:
eval average>=10
This should let you know if the average is greater than 10 jobs queued. Outlier will remove entries with absurd spikes (if you suddenly get 2000 tiny jobs that get cleared up in a few seconds, it won't throw off the average), and this condition should alert you if the average number of jobs is ever greater than 10 for a period of over two minutes.
... View more