topic Re: How do I create an alert that only fires when threshold is exceeded and resets once results are within that threshold? in Alerting

How do I create an alert that only fires when threshold is exceeded and resets once results are within that threshold?

pwilly — Tue, 29 Sep 2020 21:47:38 GMT

I have a scenario where I need to build an alert for a search that triggers on a numerical value. I need to set a threshold and alert that triggers only when that threshold is exceeded, and then have the alert reset once the value is under that threshold.

So basically, var2 returns a NUMERICAL_VALUE, which is the numerical value that I need to alert on.

For example:

when NUMERICAL_VALUE goes above 5, I want the alert to fire. So, if the NUMERICAL_VALUE goes from 4 to 6, the alert would fire, but not if it goes from 6 to 7.
If the NUMERICAL_VALUE goes from 7 back to 4, that would reset the alert. Then, if NUMERICAL_VALUE went back from 4 to 6, it would trigger the alert again.

I have the alert search set to run every two minutes.

Thanks if anyone has any answers.

index=someindex sourcetype="somesourcetype" source="somesource" Var1="StaticValue"  Var2=StaticValue | dedup VAR1 VAR2 | where NUMERICAL_VALUE  > 5

Re: How do I create an alert that only fires when threshold is exceeded and resets once results are within that threshold?

baldwintm — Wed, 31 Oct 2018 12:27:23 GMT

I would use a lookup table to keep the current state

index=someindex sourcetype="somesourcetype" source="somesource" Var1="StaticValue"  Var2=StaticValue | dedup VAR1 VAR2 | eval state=if(NUMERICAL_VALUE  > 5, 1, 0) | lookup alertstate.csv host OUTPUT state AS previous_state | outputlookup override_if_empty=false alertstate.csv | where previous_state<state

Re: How do I create an alert that only fires when threshold is exceeded and resets once results are within that threshold?

woodcock — Wed, 31 Oct 2018 19:04:58 GMT

Keep track of your alert state in a lookup table; you need 2 searches like these:

Set Alert (fire once):

index=someindex sourcetype="somesourcetype" source="somesource" VAR1="StaticValue"  VAR2="StaticValue"
| dedup VAR1 VAR2
| where NUMERICAL_VALUE  > 5
| table VAR1 VAR2 NUMERICAL_VALUE
| lookup YourLookupFileNameHere.csv VAR1 VAR2 OUTPUT fired_time
| where isnull(fired_time)
| eval fired_time=now()
| outputlookup  coverride_if_empty=false YourLookupFileNameHere.csv

Then create another scheduled search to clear out the lookup file when NUMERICAL_VALUE < 5.