The alert condition I want is based off of math comparing a potential maximum to actual usage. The result is dynamic, so a simple condition of X matches in X time is not applicable.
index=rtreport_summary source=rtreport extapp_type=LSAPI orig_host=us-portal*| stats min(jobs) AS min max(jobs) AS max mean(jobs) AS mean median(jobs) AS median p95(jobs) AS p95 stdev(jobs) AS stdev max(config_max_conn) AS config_max_conn max(req_total) AS req_total count(_raw) as samples by extapp_type hnum orig_host extapp_vhost _time| lookup host-cluster host AS orig_host| rename orig_host AS host| table _time cluster host hnum extapp_vhost min max mean median p95 stdev config_max_conn req_total samples | timechart span=1m sum(max) AS JobsInSystem sum(config_max_conn) AS MaxWorkers | eval WorkersAvailable=(MaxWorkers - JobsInSystem)
I want to alert when WorkersAvailable < 5.
