Alerting

How do I create an alert that only fires when threshold is exceeded and resets once results are within that threshold?

pwilly
Explorer

I have a scenario where I need to build an alert for a search that triggers on a numerical value. I need to set a threshold and alert that triggers only when that threshold is exceeded, and then have the alert reset once the value is under that threshold.

So basically, var2 returns a NUMERICAL_VALUE, which is the numerical value that I need to alert on.

For example:

  • when NUMERICAL_VALUE goes above 5, I want the alert to fire. So, if the NUMERICAL_VALUE goes from 4 to 6, the alert would fire, but not if it goes from 6 to 7.
  • If the NUMERICAL_VALUE goes from 7 back to 4, that would reset the alert. Then, if NUMERICAL_VALUE went back from 4 to 6, it would trigger the alert again.

I have the alert search set to run every two minutes.

Thanks if anyone has any answers.

index=someindex sourcetype="somesourcetype" source="somesource" Var1="StaticValue"  Var2=StaticValue | dedup VAR1 VAR2 | where NUMERICAL_VALUE  > 5
Tags (2)
0 Karma

woodcock
Esteemed Legend

Keep track of your alert state in a lookup table; you need 2 searches like these:

Set Alert (fire once):

index=someindex sourcetype="somesourcetype" source="somesource" VAR1="StaticValue"  VAR2="StaticValue"
| dedup VAR1 VAR2
| where NUMERICAL_VALUE  > 5
| table VAR1 VAR2 NUMERICAL_VALUE
| lookup YourLookupFileNameHere.csv VAR1 VAR2 OUTPUT fired_time
| where isnull(fired_time)
| eval fired_time=now()
| outputlookup  coverride_if_empty=false YourLookupFileNameHere.csv

Then create another scheduled search to clear out the lookup file when NUMERICAL_VALUE < 5.

baldwintm
Path Finder

I would use a lookup table to keep the current state

index=someindex sourcetype="somesourcetype" source="somesource" Var1="StaticValue"  Var2=StaticValue | dedup VAR1 VAR2 | eval state=if(NUMERICAL_VALUE  > 5, 1, 0) | lookup alertstate.csv host OUTPUT state AS previous_state | outputlookup override_if_empty=false alertstate.csv | where previous_state<state
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Splunk Auto Ingestion Parallel Pipeline Scaling

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully ...

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...