topic Re: Can an alert's actions be data-driven? in Splunk Search

Can an alert's actions be data-driven?

unitedmarsupial — Thu, 19 Dec 2019 16:24:14 GMT

Suppose, one has an alert defined for checking multiple application-instances.

Can the actions defined for the alert be different depending on the results of the search? For example, the priority of the alert's email should be High for the production instance and normal for all others.

Likewise, can the results of the search determine, whether a particular action (such as paging the sysadmin) is taken at all -- regardless of the parameters?

We'd really like to avoid having multiple copies of the same alert just so that we can have slightly different actions for each...

Re: Can an alert's actions be data-driven?

adonio — Thu, 19 Dec 2019 18:18:34 GMT

how about write the logic to the search and configure the alert accordingly?

Re: Can an alert's actions be data-driven?

unitedmarsupial — Fri, 20 Dec 2019 21:38:12 GMT

"Accordingly" is the key, is not it? Suppose, I use | eval Priority=if("Production", "High", "Normal") -- how can I make the $result.Priority$ affect that of the generated alert-email?

Re: Can an alert's actions be data-driven?

to4kawa — Sun, 22 Dec 2019 20:10:17 GMT

About alerts

Priority High > send mail
Priority Normal > nothing
Priority High > send mail and "priority high message"
Priority Normal > send mail and "priority normal message"

which one?

Re: Can an alert's actions be data-driven?

unitedmarsupial — Mon, 23 Dec 2019 16:10:40 GMT

Both would be useful, but the 1. -- especially so... Thank you, @to4kawa!

Re: Can an alert's actions be data-driven?

to4kawa — Tue, 24 Dec 2019 00:02:26 GMT

your search 
| eval Priority=if(your_Production_condition, "High", "Normal")
| where Priority="High"

Hi, @unitedmarsupials
Event count > 0 , you can run alerting and action send email.

Re: Can an alert's actions be data-driven?

unitedmarsupial — Tue, 24 Dec 2019 15:01:58 GMT

Thank you, this may be suitable in the cases, when there is only one action -- because this method determines, whether the alert fires at all. We have multiple actions -- and would like to skip some of them, but not all... Can that be done? Maybe, actions can have additional searches to append to the "base" one -- the way dashboard's panels can have?

Re: Can an alert's actions be data-driven?

woodcock — Tue, 24 Dec 2019 18:23:41 GMT

If you are talking about email, if you move to sendemail inside of your SPL and ditch the Alert Action method. You will now have complete control.

Re: Can an alert's actions be data-driven?

woodcock — Tue, 24 Dec 2019 18:28:35 GMT

Take your alert out of digest mode by setting it to For Each Result. Then you can use priority=$result.priority$ in the subject line of your email after doing something like this in your SPL:

your foundation search and other stuff here
| eval priority=case(
    condition=="bad", "High",
    condition=="really bad", "Critical",
    true(), "Low")

Re: Can an alert's actions be data-driven?

to4kawa — Wed, 25 Dec 2019 01:02:30 GMT

As @woodcock says, it's appropriate to separate priorities for each desired action.

Re: Can an alert's actions be data-driven?

woodcock — Fri, 27 Dec 2019 16:59:07 GMT

I think that what you are really looking for is this:

your foundation search and other stuff here
| eval priority=case(
     condition=="bad", "high",
     condition=="really bad", "highest",
     true(), "normal")
| outputlookup MyTempLookup.csv
| stats count by priority
| map maxsearches=5 search="|inputlookup MyTempLookup.csv
                            | search priority=$priority$
                            | sendemail priority=$priority$ to=\"who@where.com\" format=raw subject=\"My Subject\" sendresults=true"

This is based off of my answer here:
https://answers.splunk.com/answers/489475/how-configure-an-alert-to-send-an-email-based-on-f.html

Re: Can an alert's actions be data-driven?

unitedmarsupial — Thu, 02 Jan 2020 20:30:28 GMT

Ah, so sendmail is a function in itself? Is that true about all actions? For example, in addition to sending e-mail, our Splunk instance can also trigger a Moogsoft-alert -- does that mean, there is a moogsoft function too? Thank you!