Splunk Search

How to trigger and terminate the alert

chprvn
New Member

Hi,

My question is on monitoring.
Currently, we have an alert which is triggered when certain percentage of call failure happen and says the service outage.
Requirement is to send a new alert when outage is terminated(service working fine)

Any thoughts?

i thought of creating 2 alerts
Alert 1- that run every 30 min if the percentage(25%) of calls fails.
Alert 2 - to check if alert 1 is triggered in last 30 min.

When ever alert 1 is triggered, it says "Outage is happening" and it also enables alert 2. (ENABLE only if its not enabled)
Then, If Alert 1 not triggered in last 30 min, alert 2 is triggered saying "Outage is over, Service is back!".and disable the Alert 2.

Is this even possible?

0 Karma

micahkemp
Champion

I think the better solution here is to structure your search to examine the past hour, evaluate each 30 minute bin independently to determine if it should alert, and alert if the most recent 30 minute bin is in the alert state and the prior 30 minutes was not.

niketn
Legend

@chprvn, what is the query for your alert?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

chprvn
New Member

Alert 1 Query is to check the percentage of call failures
Alert 2 query is to check if alert one is triggered

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...