All Apps and Add-ons

Can Splunk be configured to use different PagerDuty escalation policies based on the alert?

pfernandez133
Explorer

Hey everyone, is there's a way to have Splunk assign alerts to different PagerDuty escalation policies based on the triggered alert?

We have different development teams, and I'd like to set it up so that if product X has an error, it alerts to the Splunk-X PagerDuty group, product Y, Splunk-Y PagerDuty group, etc.

I was thinking I could simply set up multiple PagerDuty Splunk services, but I can't think of how to configure Splunk that alert X should go to PagerDuty Splunk service "Splunk-X", while alert Y should go to PagerDuty Splunk service "Splunk-Y".

Thanks in advance!

1 Solution

pfernandez133
Explorer

Well, it's not the most elegant solution but...I created an additional Splunk service in PagerDuty, called Splunk-X. Then, on the Splunk server, I copied ~splunk/scripts/pagerduty.py to ~splunk/scripts/pagerduty-x.py. I then changed the following line in def main():

pagerduty_api_key = get_pagerduty_api_key(config_file)

to the following line:

pagerduty_api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Where xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx is the service API from Splunk-X in PagerDuty. Then, I followed PagerDuty's steps on creating an alert. Namely, you create an alert, then set it to run a script, and that script is pagerduty-x.py.

Tested. Works. Yay!

View solution in original post

davpx
Communicator

I happened to stumble across this. I know really late. Check out my fork of the old pagerduty alert app. If your alert contains a "pd_key" field with the escalation key, it will auto route to the proper service. I used it to escalate different groups of servers to different teams based on a lookup

https://github.com/davpx/splunk_app_pagerduty

0 Karma

pfernandez133
Explorer

Well, it's not the most elegant solution but...I created an additional Splunk service in PagerDuty, called Splunk-X. Then, on the Splunk server, I copied ~splunk/scripts/pagerduty.py to ~splunk/scripts/pagerduty-x.py. I then changed the following line in def main():

pagerduty_api_key = get_pagerduty_api_key(config_file)

to the following line:

pagerduty_api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Where xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx is the service API from Splunk-X in PagerDuty. Then, I followed PagerDuty's steps on creating an alert. Namely, you create an alert, then set it to run a script, and that script is pagerduty-x.py.

Tested. Works. Yay!

View solution in original post

Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.