Alerting

How do I create an alert that only fires when threshold is exceeded and resets once results are within that threshold?

pwilly
Explorer

I have a scenario where I need to build an alert for a search that triggers on a numerical value. I need to set a threshold and alert that triggers only when that threshold is exceeded, and then have the alert reset once the value is under that threshold.

So basically, var2 returns a NUMERICAL_VALUE, which is the numerical value that I need to alert on.

For example:

  • when NUMERICAL_VALUE goes above 5, I want the alert to fire. So, if the NUMERICAL_VALUE goes from 4 to 6, the alert would fire, but not if it goes from 6 to 7.
  • If the NUMERICAL_VALUE goes from 7 back to 4, that would reset the alert. Then, if NUMERICAL_VALUE went back from 4 to 6, it would trigger the alert again.

I have the alert search set to run every two minutes.

Thanks if anyone has any answers.

index=someindex sourcetype="somesourcetype" source="somesource" Var1="StaticValue"  Var2=StaticValue | dedup VAR1 VAR2 | where NUMERICAL_VALUE  > 5
Tags (2)
0 Karma

woodcock
Esteemed Legend

Keep track of your alert state in a lookup table; you need 2 searches like these:

Set Alert (fire once):

index=someindex sourcetype="somesourcetype" source="somesource" VAR1="StaticValue"  VAR2="StaticValue"
| dedup VAR1 VAR2
| where NUMERICAL_VALUE  > 5
| table VAR1 VAR2 NUMERICAL_VALUE
| lookup YourLookupFileNameHere.csv VAR1 VAR2 OUTPUT fired_time
| where isnull(fired_time)
| eval fired_time=now()
| outputlookup  coverride_if_empty=false YourLookupFileNameHere.csv

Then create another scheduled search to clear out the lookup file when NUMERICAL_VALUE < 5.

baldwintm
Path Finder

I would use a lookup table to keep the current state

index=someindex sourcetype="somesourcetype" source="somesource" Var1="StaticValue"  Var2=StaticValue | dedup VAR1 VAR2 | eval state=if(NUMERICAL_VALUE  > 5, 1, 0) | lookup alertstate.csv host OUTPUT state AS previous_state | outputlookup override_if_empty=false alertstate.csv | where previous_state<state
Get Updates on the Splunk Community!

Accelerating Observability as Code with the Splunk AI Assistant

We’ve seen in previous posts what Observability as Code (OaC) is and how it’s now essential for managing ...

Integrating Splunk Search API and Quarto to Create Reproducible Investigation ...

 Splunk is More Than Just the Web Console For Digital Forensics and Incident Response (DFIR) practitioners, ...

Congratulations to the 2025-2026 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...