https://community.splunk.com/t5/All-Apps-and-Add-ons/PagerDuty-autoresolve-of-alerts/m-p/407759#M49697 <P>Hi,</P> <P>Is it possible for splunk to resolve PD alerts after it has gone below the threshold?</P> <P>Regards<BR /> Silvano</P> Fri, 30 Nov 2018 11:48:14 GMT silvanop 2018-11-30T11:48:14Z https://community.splunk.com/t5/All-Apps-and-Add-ons/PagerDuty-autoresolve-of-alerts/m-p/407759#M49697 <P>Hi,</P> <P>Is it possible for splunk to resolve PD alerts after it has gone below the threshold?</P> <P>Regards<BR /> Silvano</P> Fri, 30 Nov 2018 11:48:14 GMT https://community.splunk.com/t5/All-Apps-and-Add-ons/PagerDuty-autoresolve-of-alerts/m-p/407759#M49697 silvanop 2018-11-30T11:48:14Z https://community.splunk.com/t5/All-Apps-and-Add-ons/PagerDuty-autoresolve-of-alerts/m-p/504275#M62120 <P>Hey Silvano.</P><P>&nbsp;</P><P>Yes there is.</P><P>You can actually trigger and resolve pagerduty's using the same alert even.</P><P>Take a look at the following example code.</P><P>*In production you would probably put this into a macro and pass the event_action as an argument...</P><P>index=_internal ERROR<BR />| stats count as event_count<BR />| eval dedup_key="ddddd"<BR />| eval severity="warning"<BR />| eval event_action=case(event_count&gt;0,"trigger",1=1,"resolve")<BR />| eval summary="A summary of this event"<BR />| eval source="a.server.example.com"<BR />| eval routing_key="SOME_ROUTING_KEY"<BR />| table dedup_key,severity,event_action, summary, source, routing_key</P><P>Basically the fields above are the minimum for a pagerduty alert.</P><P>When there is one or more results the action will be to trigger an incident, when none it will send a resolve.The dedup key will end up being the name of the search so you don't need to specify.</P><P>*note, the stats count is in case there are no results as you need something to raise an event and send a resolve. This also means this only works for a single alert.</P><P>In order for this to work you need to use event rules in pagerduty.</P><P>Creeate a new event rule and create a minimum of two rules:</P><P>-The first will be resolve. ie if result.event_action=resolve then resolve.</P><P>-The second will be trigger. is if event_action=trigger then raise an incident.</P><P>&nbsp;</P><P>There are other things you may want to do like repeat step 2 for each severity.</P><P>And that should get you auto resolving pagerduty's.</P><P>That was the best way i could find.If you found anything better since let me know.</P> Sat, 13 Jun 2020 14:07:00 GMT https://community.splunk.com/t5/All-Apps-and-Add-ons/PagerDuty-autoresolve-of-alerts/m-p/504275#M62120 jethrop 2020-06-13T14:07:00Z