Alert Recovery Notifications

[keshengjie]

I have a bunch of scheduled searches that occasionally trigger. Is there a way to have a recovery message sent if a subsequent scheduled search doesn't find any errors?

I need an easy way for our helpdesk to know if they should escalated these alerts.

[MuS · Accepted solution]

Hi keshengjie

I recently had to create something similar, where it was needed to sent an up alert but only if there was down alert before. In my case I had a field named message which was used to determine if it is up or down - So this is what I did:

YourBaseSearch 
| eval isUp = if(message=="Normal","Yes","No") 
| chart latest(eval(if(isUp=="Yes",_time,null()))) as uptime earliest(eval(if(isUp=="No",_time,null()))) as downtime by domain 
| where (isnotnull(uptime) AND ( uptime >= downtime)) 
| rename domain AS Server 
| eval Status="Up again" 
| table Server Status

This saved search runs over a 2 minutes time range and sends out an alert email if in my case the checked domain is online again. Maybe this is not exactly what you asked for, but could give you a hint on how to create up alerts only if there was a down alert before.

cheers, MuS