All Apps and Add-ons

PagerDuty autoresolve of alerts

silvanop
New Member

Hi,

Is it possible for splunk to resolve PD alerts after it has gone below the threshold?

Regards
Silvano

0 Karma

jethrop
Explorer

Hey Silvano.

 

Yes there is.

You can actually trigger and resolve pagerduty's using the same alert even.

Take a look at the following example code.

*In production you would probably put this into a macro and pass the event_action as an argument...

index=_internal ERROR
| stats count as event_count
| eval dedup_key="ddddd"
| eval severity="warning"
| eval event_action=case(event_count>0,"trigger",1=1,"resolve")
| eval summary="A summary of this event"
| eval source="a.server.example.com"
| eval routing_key="SOME_ROUTING_KEY"
| table dedup_key,severity,event_action, summary, source, routing_key

Basically the fields above are the minimum for a pagerduty alert.

When there is one or more results the action will be to trigger an incident, when none it will send a resolve.The dedup key will end up being the name of the search so you don't need to specify.

*note, the stats count is in case there are no results as you need something to raise an event and send a resolve. This also means this only works for a single alert.

In order for this to work you need to use event rules in pagerduty.

Creeate a new event rule and create a minimum of two rules:

-The first will be resolve. ie if result.event_action=resolve then resolve.

-The second will be trigger. is if event_action=trigger then raise an incident.

 

There are other things you may want to do like repeat step 2 for each severity.

And that should get you auto resolving pagerduty's.

That was the best way i could find.If you found anything better since let me know.

0 Karma
Get Updates on the Splunk Community!

What’s New & Next in Splunk SOAR

Security teams today are dealing with more alerts, more tools, and more pressure than ever.  Join us on ...

Your Voice Matters! Help Us Shape the New Splunk Lantern Experience

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

September Community Champions: A Shoutout to Our Contributors!

As we close the books on another fantastic month, we want to take a moment to celebrate the people who are the ...