Alerting

Conditional Alerting help

hoopydave
Path Finder

Hey all. I'm using Splunk 6.4.10. My search is:
index=myindex host=myhost result error code 100
Trigger Condition: Number of Results is > 0
The Cron schedule is every 5th minute

I'm expecting the results should be 0. I only want to be alerted if the trigger condition is met three times in succession. Many times the condition will correct itself, so I don't want an alert every time, only after the third time.

So to summarize:
I need to setup a conditional alert with the following criteria:
The number of results returned from my search must be greater than 0
Only generate an alert if the trigger condition happens three times in succession. If at any time the results go back to 0 in that time frame, the count resets.

0 Karma

dmarling
Builder

It sounds like you only want it to alert if there are three consecutive 5 minute buckets of > 0 results, which would mean we would only need to adjust the query and you can leave everything else the same:

index=myindex host=myhost result error code 100 earliest=-15m latest=now 
| timechart span=5m count 
| search count>0
| stats count 
| search count=3

The above query is now performing a 15 minute look back and performing a count of results from the last 3 5 minute buckets. The subsequent search is filtering out the ones that had zero results. The last stats is counting the amount of rows after that filter and the final search will only return results to trigger an alert if the count equals 3. That should satisfy your requirements.

If this comment/answer was helpful, please up vote it. Thank you.
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...

SplunkTrust Application Period is Officially OPEN!

It's that time, folks! The application/nomination period for the 2026-2027 SplunkTrust is officially open. If ...