All Apps and Add-ons

PagerDuty autoresolve of alerts

silvanop
New Member

Hi,

Is it possible for splunk to resolve PD alerts after it has gone below the threshold?

Regards
Silvano

0 Karma

jethrop
Explorer

Hey Silvano.

 

Yes there is.

You can actually trigger and resolve pagerduty's using the same alert even.

Take a look at the following example code.

*In production you would probably put this into a macro and pass the event_action as an argument...

index=_internal ERROR
| stats count as event_count
| eval dedup_key="ddddd"
| eval severity="warning"
| eval event_action=case(event_count>0,"trigger",1=1,"resolve")
| eval summary="A summary of this event"
| eval source="a.server.example.com"
| eval routing_key="SOME_ROUTING_KEY"
| table dedup_key,severity,event_action, summary, source, routing_key

Basically the fields above are the minimum for a pagerduty alert.

When there is one or more results the action will be to trigger an incident, when none it will send a resolve.The dedup key will end up being the name of the search so you don't need to specify.

*note, the stats count is in case there are no results as you need something to raise an event and send a resolve. This also means this only works for a single alert.

In order for this to work you need to use event rules in pagerduty.

Creeate a new event rule and create a minimum of two rules:

-The first will be resolve. ie if result.event_action=resolve then resolve.

-The second will be trigger. is if event_action=trigger then raise an incident.

 

There are other things you may want to do like repeat step 2 for each severity.

And that should get you auto resolving pagerduty's.

That was the best way i could find.If you found anything better since let me know.

0 Karma
Get Updates on the Splunk Community!

Join Us for Splunk University and Get Your Bootcamp Game On!

If you know, you know! Splunk University is the vibe this summer so register today for bootcamps galore ...

.conf24 | Learning Tracks for Security, Observability, Platform, and Developers!

.conf24 is taking place at The Venetian in Las Vegas from June 11 - 14. Continue reading to learn about the ...

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...