Can Splunk be configured to use different PagerDuty escalation policies based on the alert?

[pfernandez133]

Hey everyone, is there's a way to have Splunk assign alerts to different PagerDuty escalation policies based on the triggered alert?

We have different development teams, and I'd like to set it up so that if product X has an error, it alerts to the Splunk-X PagerDuty group, product Y, Splunk-Y PagerDuty group, etc.

I was thinking I could simply set up multiple PagerDuty Splunk services, but I can't think of how to configure Splunk that alert X should go to PagerDuty Splunk service "Splunk-X", while alert Y should go to PagerDuty Splunk service "Splunk-Y".

Thanks in advance!

[davpx]

I happened to stumble across this. I know really late. Check out my fork of the old pagerduty alert app. If your alert contains a "pd_key" field with the escalation key, it will auto route to the proper service. I used it to escalate different groups of servers to different teams based on a lookup

https://github.com/davpx/splunk_app_pagerduty

[pfernandez133 · Accepted solution]

Well, it's not the most elegant solution but...I created an additional Splunk service in PagerDuty, called Splunk-X. Then, on the Splunk server, I copied ~splunk/scripts/pagerduty.py to ~splunk/scripts/pagerduty-x.py. I then changed the following line in def main():

pagerduty_api_key = get_pagerduty_api_key(config_file)

to the following line:

pagerduty_api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Where xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx is the service API from Splunk-X in PagerDuty. Then, I followed PagerDuty's steps on creating an alert. Namely, you create an alert, then set it to run a script, and that script is pagerduty-x.py.

Tested. Works. Yay!