Alerting

How do I create an alert that only fires when threshold is exceeded and resets once results are within that threshold?

pwilly
Explorer

I have a scenario where I need to build an alert for a search that triggers on a numerical value. I need to set a threshold and alert that triggers only when that threshold is exceeded, and then have the alert reset once the value is under that threshold.

So basically, var2 returns a NUMERICAL_VALUE, which is the numerical value that I need to alert on.

For example:

  • when NUMERICAL_VALUE goes above 5, I want the alert to fire. So, if the NUMERICAL_VALUE goes from 4 to 6, the alert would fire, but not if it goes from 6 to 7.
  • If the NUMERICAL_VALUE goes from 7 back to 4, that would reset the alert. Then, if NUMERICAL_VALUE went back from 4 to 6, it would trigger the alert again.

I have the alert search set to run every two minutes.

Thanks if anyone has any answers.

index=someindex sourcetype="somesourcetype" source="somesource" Var1="StaticValue"  Var2=StaticValue | dedup VAR1 VAR2 | where NUMERICAL_VALUE  > 5
Tags (2)
0 Karma

woodcock
Esteemed Legend

Keep track of your alert state in a lookup table; you need 2 searches like these:

Set Alert (fire once):

index=someindex sourcetype="somesourcetype" source="somesource" VAR1="StaticValue"  VAR2="StaticValue"
| dedup VAR1 VAR2
| where NUMERICAL_VALUE  > 5
| table VAR1 VAR2 NUMERICAL_VALUE
| lookup YourLookupFileNameHere.csv VAR1 VAR2 OUTPUT fired_time
| where isnull(fired_time)
| eval fired_time=now()
| outputlookup  coverride_if_empty=false YourLookupFileNameHere.csv

Then create another scheduled search to clear out the lookup file when NUMERICAL_VALUE < 5.

baldwintm
Path Finder

I would use a lookup table to keep the current state

index=someindex sourcetype="somesourcetype" source="somesource" Var1="StaticValue"  Var2=StaticValue | dedup VAR1 VAR2 | eval state=if(NUMERICAL_VALUE  > 5, 1, 0) | lookup alertstate.csv host OUTPUT state AS previous_state | outputlookup override_if_empty=false alertstate.csv | where previous_state<state
Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Dynamic formatting from XML events

This challenge was first posted on Slack #puzzles channelFor a previous puzzle, I needed a set of fixed-length ...

Enter the Agentic Era with Splunk AI Assistant for SPL 1.4

  &#x1f680; Your data just got a serious AI upgrade — are you ready? Say hello to the Agentic Era with the ...

Stronger Security with Federated Search for S3, GCP SQL & Australian Threat ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...