Splunk Search

Can an alert's actions be data-driven?

unitedmarsupial
Path Finder

Suppose, one has an alert defined for checking multiple application-instances.

Can the actions defined for the alert be different depending on the results of the search? For example, the priority of the alert's email should be High for the production instance and normal for all others.

Likewise, can the results of the search determine, whether a particular action (such as paging the sysadmin) is taken at all -- regardless of the parameters?

We'd really like to avoid having multiple copies of the same alert just so that we can have slightly different actions for each...

Tags (2)
0 Karma

woodcock
Esteemed Legend

I think that what you are really looking for is this:

your foundation search and other stuff here
| eval priority=case(
     condition=="bad", "high",
     condition=="really bad", "highest",
     true(), "normal")
| outputlookup MyTempLookup.csv
| stats count by priority
| map maxsearches=5 search="|inputlookup MyTempLookup.csv
                            | search priority=$priority$
                            | sendemail priority=$priority$ to=\"who@where.com\" format=raw subject=\"My Subject\" sendresults=true"

This is based off of my answer here:
https://answers.splunk.com/answers/489475/how-configure-an-alert-to-send-an-email-based-on-f.html

0 Karma

unitedmarsupial
Path Finder

Ah, so sendmail is a function in itself? Is that true about all actions? For example, in addition to sending e-mail, our Splunk instance can also trigger a Moogsoft-alert -- does that mean, there is a moogsoft function too? Thank you!

0 Karma

woodcock
Esteemed Legend

Take your alert out of digest mode by setting it to For Each Result. Then you can use priority=$result.priority$ in the subject line of your email after doing something like this in your SPL:

your foundation search and other stuff here
| eval priority=case(
    condition=="bad", "High",
    condition=="really bad", "Critical",
    true(), "Low")

woodcock
Esteemed Legend

If you are talking about email, if you move to sendemail inside of your SPL and ditch the Alert Action method. You will now have complete control.

0 Karma

to4kawa
Ultra Champion
your search 
| eval Priority=if(your_Production_condition, "High", "Normal")
| where Priority="High"

Hi, @unitedmarsupials
Event count > 0 , you can run alerting and action send email.

0 Karma

unitedmarsupial
Path Finder

Thank you, this may be suitable in the cases, when there is only one action -- because this method determines, whether the alert fires at all. We have multiple actions -- and would like to skip some of them, but not all... Can that be done? Maybe, actions can have additional searches to append to the "base" one -- the way dashboard's panels can have?

0 Karma

to4kawa
Ultra Champion

As @woodcock says, it's appropriate to separate priorities for each desired action.

0 Karma

adonio
Ultra Champion

how about write the logic to the search and configure the alert accordingly?

0 Karma

unitedmarsupial
Path Finder

"Accordingly" is the key, is not it? Suppose, I use | eval Priority=if("Production", "High", "Normal") -- how can I make the $result.Priority$ affect that of the generated alert-email?

0 Karma

to4kawa
Ultra Champion

About alerts

  1. Priority High > send mail
    Priority Normal > nothing

  2. Priority High > send mail and "priority high message"
    Priority Normal > send mail and "priority normal message"

which one?

0 Karma

unitedmarsupial
Path Finder

Both would be useful, but the 1. -- especially so... Thank you, @to4kawa!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...