Alerting

escalate an alert to a different email address after some time or a certain frequency of its occurrence

mmaqbool
Explorer

Hi,

I am new to splunk and working on imitating nagaios's alerting behavior.

Specifically, I have created an alert that emails user A when a certain event happens. I now want to stop sending the email to user A if:

1-For eg the event has occurred 5 times in a minuit.
2-For eg the event has occurred 5 times (regardless of the time it took for 5 occurrences).
3-For eg the event has occurred again within 5 minuits of the previous occurrence.

And now send the email to user C and D.

In simple words, escalating an alert to other teams based on some criteria.

Looking forward for some help in this regard.

Kind regards,
mmaqbool.

Tags (1)
0 Karma

woodcock
Esteemed Legend

The general approach is to have 2 searches. The main search dumps out triggers into a lookup file using outputlookup append=t.
Another search runs every minute and examines the triggers in the file and applies the logic that you describe to decide whether and to whom to send an email.

0 Karma

vik_splunk
Communicator

Hi @mmaqbool

Out of the box, I don't think this is a supported feature to the best of my knowledge.

I do however think that the below approach can be used to achieve the same.

1)Do not hard code the e-mail address but try to leverage a field from the search results by tokenizing the e-mail address field. This can be achieved by constructing a hidden e-mail address field(any field name prefixed with an _ is automatically hidden in the output but can be used as tokens in the e-mail alert parameters.)

2)Use _internal data and specifically the scheduler logs to figure out which instance of alert this is. So ideally you will use an append and scheduler data to construct your e-mail address field.

As a small SPL example of how this would work,

"basesearch"|append [search index=_internal sourcetype=scheduler alert_actions=email savedsearch_name="alertname"|stats count(eval(if(isnull(_raw),0,_raw))) AS Count|eval _emailaddress=case(Count=0,"emailaddress1",Count=1,"emailaddress2",Count>2,"emailaddress3")|fields _emailaddress]

"To" field of the alert to be set as $result._emailaddress$

Please do upvote if the approach above works for you!

mmaqbool
Explorer

Dear vik_splunk,

Thankyou so much for the help.

I am actually quite new to splunk therefore I am having troubles understanding your approach. Could you exposit it in a bit non-technical language?

For now, I am trying to understand what the SPL expression is doing. I don't understand the following on the face of it:

1-first pipe just before append. What are you piping, the original event search query on which the alert is firing?

2-The append command runs only over historical data and does not produce correct results if used in a real-time search. [https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Append]

3-savedsearch_name=|, you didn't specify any savedsearch_name instead put a pipe in front of it again.

4-stats count(eval(if(isnull(_raw),0,_raw))) AS Count, this just flew right over my head.

5-eval _emailaddress=case(Count=0,"",Count=1,"",Count>2,"")|fields _emailaddress, this seems like the part where based on the number or frequency of a particular alert you are emailing different people, right?

6-one last question. what will happen to the original alert? is this an "alert on top of alerts" thing? [a similar approach: https://answers.splunk.com/answers/237950/triggering-an-alert-on-alerts-alert-on-alerts.html] or will the original alert stop firing to user A, say, and now is only going to the inbox of B or C etc?

Many thanks for your time.
mmaqbool

0 Karma

vik_splunk
Communicator

I see where the confusion is stemming from @mmaqbool

I've since edited the original post to show text in double quotes as it had removed all content I had enclosed within <>.

To answer your questions
1)Yes you need to pipe your base search to this using append
2)Yes that's correct. Since you will be scheduling your search(as opposed to real-time), append works just fine.
3)You need to mention the name of the search being executed.(edited original post)
4)Stats - It just attempting to count the number of alert e-mails sent (you need to use an earliest and latest within the append operation to indicate the time period to check). But obviously stats returns nothing if no events are found. We need stats to return 0 if no events are found. that's what the if and eval are doing.
5)You are right in your interpretation
6)No. This is the alert configuration. This is not an alert of alerts type thing. We are just adding a conditional logic to identify the appropriate e-mail address to send the alert to.

Hope that helps.

vik_splunk
Communicator

@mmaqbool did you try the above? did it work?

0 Karma

mmaqbool
Explorer

Dear vik_splunk,

Thankyou so much for the time you took to exposit such an elaborate reply to my questions. My manager said that escalation is not a critical concern right now, and assigned me to something else; which is why I could not find the time to implement your solution as such. But when the time comes, hopefully soon, then I will definitely implement it and let you know. Or, if someone else is grappling with the same issue and by chance lands upon this thread before myself, then maybe they could let us know whether it works.

Thanks a lot once again for the help mate, much appreciated:)

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

@vik_splunk keep in mind that 6.1 is about to reach its end of life - probably not a good idea to keep linking to that version's docs.

0 Karma

vik_splunk
Communicator

removed the url @martin_mueller

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

No need to remove it, you could just link to a recent version.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...