All Apps and Add-ons

Can Splunk be configured to use different PagerDuty escalation policies based on the alert?

pfernandez133
Explorer

Hey everyone, is there's a way to have Splunk assign alerts to different PagerDuty escalation policies based on the triggered alert?

We have different development teams, and I'd like to set it up so that if product X has an error, it alerts to the Splunk-X PagerDuty group, product Y, Splunk-Y PagerDuty group, etc.

I was thinking I could simply set up multiple PagerDuty Splunk services, but I can't think of how to configure Splunk that alert X should go to PagerDuty Splunk service "Splunk-X", while alert Y should go to PagerDuty Splunk service "Splunk-Y".

Thanks in advance!

1 Solution

pfernandez133
Explorer

Well, it's not the most elegant solution but...I created an additional Splunk service in PagerDuty, called Splunk-X. Then, on the Splunk server, I copied ~splunk/scripts/pagerduty.py to ~splunk/scripts/pagerduty-x.py. I then changed the following line in def main():

pagerduty_api_key = get_pagerduty_api_key(config_file)

to the following line:

pagerduty_api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Where xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx is the service API from Splunk-X in PagerDuty. Then, I followed PagerDuty's steps on creating an alert. Namely, you create an alert, then set it to run a script, and that script is pagerduty-x.py.

Tested. Works. Yay!

View solution in original post

davpx
Communicator

I happened to stumble across this. I know really late. Check out my fork of the old pagerduty alert app. If your alert contains a "pd_key" field with the escalation key, it will auto route to the proper service. I used it to escalate different groups of servers to different teams based on a lookup

https://github.com/davpx/splunk_app_pagerduty

0 Karma

pfernandez133
Explorer

Well, it's not the most elegant solution but...I created an additional Splunk service in PagerDuty, called Splunk-X. Then, on the Splunk server, I copied ~splunk/scripts/pagerduty.py to ~splunk/scripts/pagerduty-x.py. I then changed the following line in def main():

pagerduty_api_key = get_pagerduty_api_key(config_file)

to the following line:

pagerduty_api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Where xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx is the service API from Splunk-X in PagerDuty. Then, I followed PagerDuty's steps on creating an alert. Namely, you create an alert, then set it to run a script, and that script is pagerduty-x.py.

Tested. Works. Yay!

Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...