All Apps and Add-ons

PagerDuty autoresolve of alerts

silvanop
New Member

Hi,

Is it possible for splunk to resolve PD alerts after it has gone below the threshold?

Regards
Silvano

0 Karma

jethrop
Explorer

Hey Silvano.

 

Yes there is.

You can actually trigger and resolve pagerduty's using the same alert even.

Take a look at the following example code.

*In production you would probably put this into a macro and pass the event_action as an argument...

index=_internal ERROR
| stats count as event_count
| eval dedup_key="ddddd"
| eval severity="warning"
| eval event_action=case(event_count>0,"trigger",1=1,"resolve")
| eval summary="A summary of this event"
| eval source="a.server.example.com"
| eval routing_key="SOME_ROUTING_KEY"
| table dedup_key,severity,event_action, summary, source, routing_key

Basically the fields above are the minimum for a pagerduty alert.

When there is one or more results the action will be to trigger an incident, when none it will send a resolve.The dedup key will end up being the name of the search so you don't need to specify.

*note, the stats count is in case there are no results as you need something to raise an event and send a resolve. This also means this only works for a single alert.

In order for this to work you need to use event rules in pagerduty.

Creeate a new event rule and create a minimum of two rules:

-The first will be resolve. ie if result.event_action=resolve then resolve.

-The second will be trigger. is if event_action=trigger then raise an incident.

 

There are other things you may want to do like repeat step 2 for each severity.

And that should get you auto resolving pagerduty's.

That was the best way i could find.If you found anything better since let me know.

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...

Secure Your Future: Mastering Upgrade Readiness for Splunk 10

Spotlight: The Splunk Health Assistant Add-On  The Splunk Health Assistant Add-On is your ultimate companion ...

Observability Unlocked: Kubernetes & Cloud Monitoring with Splunk IM

Ready to master Kubernetes and cloud monitoring like the pros? Join Splunk’s Growth Engineering team on ...