Splunk Search

Can an alert's actions be data-driven?

Path Finder

Suppose, one has an alert defined for checking multiple application-instances.

Can the actions defined for the alert be different depending on the results of the search? For example, the priority of the alert's email should be High for the production instance and normal for all others.

Likewise, can the results of the search determine, whether a particular action (such as paging the sysadmin) is taken at all -- regardless of the parameters?

We'd really like to avoid having multiple copies of the same alert just so that we can have slightly different actions for each...

Tags (2)
0 Karma

Esteemed Legend

I think that what you are really looking for is this:

your foundation search and other stuff here
| eval priority=case(
     condition=="bad", "high",
     condition=="really bad", "highest",
     true(), "normal")
| outputlookup MyTempLookup.csv
| stats count by priority
| map maxsearches=5 search="|inputlookup MyTempLookup.csv
                            | search priority=$priority$
                            | sendemail priority=$priority$ to=\"who@where.com\" format=raw subject=\"My Subject\" sendresults=true"

This is based off of my answer here:
https://answers.splunk.com/answers/489475/how-configure-an-alert-to-send-an-email-based-on-f.html

0 Karma

Path Finder

Ah, so sendmail is a function in itself? Is that true about all actions? For example, in addition to sending e-mail, our Splunk instance can also trigger a Moogsoft-alert -- does that mean, there is a moogsoft function too? Thank you!

0 Karma

Esteemed Legend

Take your alert out of digest mode by setting it to For Each Result. Then you can use priority=$result.priority$ in the subject line of your email after doing something like this in your SPL:

your foundation search and other stuff here
| eval priority=case(
    condition=="bad", "High",
    condition=="really bad", "Critical",
    true(), "Low")

Esteemed Legend

If you are talking about email, if you move to sendemail inside of your SPL and ditch the Alert Action method. You will now have complete control.

0 Karma

Ultra Champion
your search 
| eval Priority=if(your_Production_condition, "High", "Normal")
| where Priority="High"

Hi, @unitedmarsupials
Event count > 0 , you can run alerting and action send email.

0 Karma

Path Finder

Thank you, this may be suitable in the cases, when there is only one action -- because this method determines, whether the alert fires at all. We have multiple actions -- and would like to skip some of them, but not all... Can that be done? Maybe, actions can have additional searches to append to the "base" one -- the way dashboard's panels can have?

0 Karma

Ultra Champion

As @woodcock says, it's appropriate to separate priorities for each desired action.

0 Karma

SplunkTrust
SplunkTrust

how about write the logic to the search and configure the alert accordingly?

0 Karma

Path Finder

"Accordingly" is the key, is not it? Suppose, I use | eval Priority=if("Production", "High", "Normal") -- how can I make the $result.Priority$ affect that of the generated alert-email?

0 Karma

Ultra Champion

About alerts

  1. Priority High > send mail
    Priority Normal > nothing

  2. Priority High > send mail and "priority high message"
    Priority Normal > send mail and "priority normal message"

which one?

0 Karma

Path Finder

Both would be useful, but the 1. -- especially so... Thank you, @to4kawa!

0 Karma