All Apps and Add-ons

Can Splunk be configured to use different PagerDuty escalation policies based on the alert?

pfernandez133
Explorer

Hey everyone, is there's a way to have Splunk assign alerts to different PagerDuty escalation policies based on the triggered alert?

We have different development teams, and I'd like to set it up so that if product X has an error, it alerts to the Splunk-X PagerDuty group, product Y, Splunk-Y PagerDuty group, etc.

I was thinking I could simply set up multiple PagerDuty Splunk services, but I can't think of how to configure Splunk that alert X should go to PagerDuty Splunk service "Splunk-X", while alert Y should go to PagerDuty Splunk service "Splunk-Y".

Thanks in advance!

1 Solution

pfernandez133
Explorer

Well, it's not the most elegant solution but...I created an additional Splunk service in PagerDuty, called Splunk-X. Then, on the Splunk server, I copied ~splunk/scripts/pagerduty.py to ~splunk/scripts/pagerduty-x.py. I then changed the following line in def main():

pagerduty_api_key = get_pagerduty_api_key(config_file)

to the following line:

pagerduty_api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Where xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx is the service API from Splunk-X in PagerDuty. Then, I followed PagerDuty's steps on creating an alert. Namely, you create an alert, then set it to run a script, and that script is pagerduty-x.py.

Tested. Works. Yay!

View solution in original post

davpx
Communicator

I happened to stumble across this. I know really late. Check out my fork of the old pagerduty alert app. If your alert contains a "pd_key" field with the escalation key, it will auto route to the proper service. I used it to escalate different groups of servers to different teams based on a lookup

https://github.com/davpx/splunk_app_pagerduty

0 Karma

pfernandez133
Explorer

Well, it's not the most elegant solution but...I created an additional Splunk service in PagerDuty, called Splunk-X. Then, on the Splunk server, I copied ~splunk/scripts/pagerduty.py to ~splunk/scripts/pagerduty-x.py. I then changed the following line in def main():

pagerduty_api_key = get_pagerduty_api_key(config_file)

to the following line:

pagerduty_api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Where xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx is the service API from Splunk-X in PagerDuty. Then, I followed PagerDuty's steps on creating an alert. Namely, you create an alert, then set it to run a script, and that script is pagerduty-x.py.

Tested. Works. Yay!

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...