The alert condition I want is based off of math comparing a potential maximum to actual usage. The result is dynamic, so a simple condition of X matches in X time is not applicable.
index=rtreport_summary source=rtreport extapp_type=LSAPI orig_host=us-portal*| stats min(jobs) AS min max(jobs) AS max mean(jobs) AS mean median(jobs) AS median p95(jobs) AS p95 stdev(jobs) AS stdev max(config_max_conn) AS config_max_conn max(req_total) AS req_total count(_raw) as samples by extapp_type hnum orig_host extapp_vhost _time| lookup host-cluster host AS orig_host| rename orig_host AS host| table _time cluster host hnum extapp_vhost min max mean median p95 stdev config_max_conn req_total samples | timechart span=1m sum(max) AS JobsInSystem sum(config_max_conn) AS MaxWorkers | eval WorkersAvailable=(MaxWorkers - JobsInSystem)
I want to alert when WorkersAvailable < 5.
How about adding a where clause " | where WorkersAvailable < 5
" in the end of the search and set your alert go off when number of events by the search greater than 0?
How about adding a where clause " | where WorkersAvailable < 5
" in the end of the search and set your alert go off when number of events by the search greater than 0?
Beautiful! That works properly. Please excuse my ignorance, I've got a lot to learn about this software.