All Apps and Add-ons

PagerDuty autoresolve of alerts

silvanop
New Member

Hi,

Is it possible for splunk to resolve PD alerts after it has gone below the threshold?

Regards
Silvano

0 Karma

jethrop
Explorer

Hey Silvano.

 

Yes there is.

You can actually trigger and resolve pagerduty's using the same alert even.

Take a look at the following example code.

*In production you would probably put this into a macro and pass the event_action as an argument...

index=_internal ERROR
| stats count as event_count
| eval dedup_key="ddddd"
| eval severity="warning"
| eval event_action=case(event_count>0,"trigger",1=1,"resolve")
| eval summary="A summary of this event"
| eval source="a.server.example.com"
| eval routing_key="SOME_ROUTING_KEY"
| table dedup_key,severity,event_action, summary, source, routing_key

Basically the fields above are the minimum for a pagerduty alert.

When there is one or more results the action will be to trigger an incident, when none it will send a resolve.The dedup key will end up being the name of the search so you don't need to specify.

*note, the stats count is in case there are no results as you need something to raise an event and send a resolve. This also means this only works for a single alert.

In order for this to work you need to use event rules in pagerduty.

Creeate a new event rule and create a minimum of two rules:

-The first will be resolve. ie if result.event_action=resolve then resolve.

-The second will be trigger. is if event_action=trigger then raise an incident.

 

There are other things you may want to do like repeat step 2 for each severity.

And that should get you auto resolving pagerduty's.

That was the best way i could find.If you found anything better since let me know.

0 Karma
Get Updates on the Splunk Community!

Splunk Search APIを使えば調査過程が残せます

   このゲストブログは、JCOM株式会社の情報セキュリティ本部・専任部長である渡辺慎太郎氏によって執筆されました。 Note: This article is published in both Japanese ...

Integrating Splunk Search API and Quarto to Create Reproducible Investigation ...

 Splunk is More Than Just the Web Console For Digital Forensics and Incident Response (DFIR) practitioners, ...

Congratulations to the 2025-2026 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...