Is there is a way to have a scheduled search run and only alert if a certain condition is met for a period of time?
We have a search that looks for a delay value that we are extracting with rex. We want to be alerted if this value is > 1 minute for a period of time. (e.g. if the value for this delay filed is greater > 1 minute for 10 minutes, send an alert)
Is this possible with the current monitoring and alerting capabilities? I only see number of events, hosts etc as a conditional alert.
There is a conditional alert, but I suppose you are looking for a value such that the condition is met over a period of time, i.e., you want to refer to a history of running the search on a schedule. The easiest way to do this is to just run a search over a longer period of time and count (e.g., ... | bucket _time span=1min | ... by _time | stats count) but this can be very inefficient. A better way is to use a pair of searches, one to update a lookup table containing the history over time, and the second to report/alert on the lookup table.