I am new to splunk and working on imitating nagaios's alerting behavior.
Specifically, I have created an alert that emails user A when a certain event happens. I now want to stop sending the email to user A if:
1-For eg the event has occurred 5 times in a minuit.
2-For eg the event has occurred 5 times (regardless of the time it took for 5 occurrences).
3-For eg the event has occurred again within 5 minuits of the previous occurrence.
And now send the email to user C and D.
In simple words, escalating an alert to other teams based on some criteria.
Looking forward for some help in this regard.
The general approach is to have 2 searches. The main search dumps out triggers into a lookup file using
Another search runs every minute and examines the triggers in the file and applies the logic that you describe to decide whether and to whom to send an email.
Out of the box, I don't think this is a supported feature to the best of my knowledge.
I do however think that the below approach can be used to achieve the same.
1)Do not hard code the e-mail address but try to leverage a field from the search results by tokenizing the e-mail address field. This can be achieved by constructing a hidden e-mail address field(any field name prefixed with an _ is automatically hidden in the output but can be used as tokens in the e-mail alert parameters.)
2)Use _internal data and specifically the scheduler logs to figure out which instance of alert this is. So ideally you will use an append and scheduler data to construct your e-mail address field.
As a small SPL example of how this would work,
"basesearch"|append [search index=internal sourcetype=scheduler alertactions=email savedsearchname="alertname"|stats count(eval(if(isnull(raw),0,_raw))) AS Count|eval _emailaddress=case(Count=0,"emailaddress1",Count=1,"emailaddress2",Count>2,"emailaddress3")|fields _emailaddress]
"To" field of the alert to be set as $result._emailaddress$
Please do upvote if the approach above works for you!
Thankyou so much for the help.
I am actually quite new to splunk therefore I am having troubles understanding your approach. Could you exposit it in a bit non-technical language?
For now, I am trying to understand what the SPL expression is doing. I don't understand the following on the face of it:
1-first pipe just before append. What are you piping, the original event search query on which the alert is firing?
2-The append command runs only over historical data and does not produce correct results if used in a real-time search. [https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Append]
3-savedsearchname=|, you didn't specify any savedsearchname instead put a pipe in front of it again.
4-stats count(eval(if(isnull(raw),0,raw))) AS Count, this just flew right over my head.
5-eval _emailaddress=case(Count=0,"",Count=1,"",Count>2,"")|fields _emailaddress, this seems like the part where based on the number or frequency of a particular alert you are emailing different people, right?
6-one last question. what will happen to the original alert? is this an "alert on top of alerts" thing? [a similar approach: https://answers.splunk.com/answers/237950/triggering-an-alert-on-alerts-alert-on-alerts.html] or will the original alert stop firing to user A, say, and now is only going to the inbox of B or C etc?
Many thanks for your time.
I see where the confusion is stemming from @mmaqbool
I've since edited the original post to show text in double quotes as it had removed all content I had enclosed within <>.
To answer your questions
1)Yes you need to pipe your base search to this using append
2)Yes that's correct. Since you will be scheduling your search(as opposed to real-time), append works just fine.
3)You need to mention the name of the search being executed.(edited original post)
4)Stats - It just attempting to count the number of alert e-mails sent (you need to use an earliest and latest within the append operation to indicate the time period to check). But obviously stats returns nothing if no events are found. We need stats to return 0 if no events are found. that's what the if and eval are doing.
5)You are right in your interpretation
6)No. This is the alert configuration. This is not an alert of alerts type thing. We are just adding a conditional logic to identify the appropriate e-mail address to send the alert to.
Hope that helps.
Thankyou so much for the time you took to exposit such an elaborate reply to my questions. My manager said that escalation is not a critical concern right now, and assigned me to something else; which is why I could not find the time to implement your solution as such. But when the time comes, hopefully soon, then I will definitely implement it and let you know. Or, if someone else is grappling with the same issue and by chance lands upon this thread before myself, then maybe they could let us know whether it works.
Thanks a lot once again for the help mate, much appreciated:)