I'm able to send an email, when a condition is met, and also to setup the next time the alert has to be exectuted after the first match... But is it possible to send an email to inform that the alert is ok (values are normal again...) ?
add the condition in the search, instead of the alert condition.
mysearch | eval alert_result=if(condition=true,"status_alert","status_ok")
and send the alert every time with "always" , then you do the triage based on this field "alert_result".
What I missing in Splunk with your query, is a second condition for throttling, so I could configure :
alert condition : alert_result=true -> send alert email
execute search every X Minutes, until condition2 is met (alert_result=status_ok), and send email "alert ok"...
That's an interesting solution, but I that case, I get an alert everytime the search is run.
A proper alerting would be something like that
When there is no error, run the search once per hour
If alert is raised then
- send a alert message
- rerun the query maybe more often (every 15min)
- send a message when it is ok again, and run query with normal interval again...
Thank you for your proposition.
I really need an "ok email", because all our other alerting (outside from splunk) work like this.
I've understood you workaround, and I guess it will be working, it only a lot of handwork to make 2 alerts per query...
I found a duplicate, were it is described. Many thanks.
I hope Splunk will add such a feature in there wishlist for next release.
sbsbb - we did this in a different way using python code to check a table on the 'shall I notify question' e.g. do I already know about it (a question you didn't ask, I guess, but could be an issue)..but all of that was outside of Splunk's email function. To keep it inside there is no reason why you couldn't create a lookup to see if a) you already know about it...and b) figure that people may like to know its ok now. A little more work on the notification qualification, but may achieve what you seek?
I'm not posting as an answer, because its just a potential , others may have something.