All Apps and Add-ons

Can Splunk be configured to use different PagerDuty escalation policies based on the alert?

pfernandez133
Explorer

Hey everyone, is there's a way to have Splunk assign alerts to different PagerDuty escalation policies based on the triggered alert?

We have different development teams, and I'd like to set it up so that if product X has an error, it alerts to the Splunk-X PagerDuty group, product Y, Splunk-Y PagerDuty group, etc.

I was thinking I could simply set up multiple PagerDuty Splunk services, but I can't think of how to configure Splunk that alert X should go to PagerDuty Splunk service "Splunk-X", while alert Y should go to PagerDuty Splunk service "Splunk-Y".

Thanks in advance!

1 Solution

pfernandez133
Explorer

Well, it's not the most elegant solution but...I created an additional Splunk service in PagerDuty, called Splunk-X. Then, on the Splunk server, I copied ~splunk/scripts/pagerduty.py to ~splunk/scripts/pagerduty-x.py. I then changed the following line in def main():

pagerduty_api_key = get_pagerduty_api_key(config_file)

to the following line:

pagerduty_api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Where xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx is the service API from Splunk-X in PagerDuty. Then, I followed PagerDuty's steps on creating an alert. Namely, you create an alert, then set it to run a script, and that script is pagerduty-x.py.

Tested. Works. Yay!

View solution in original post

davpx
Communicator

I happened to stumble across this. I know really late. Check out my fork of the old pagerduty alert app. If your alert contains a "pd_key" field with the escalation key, it will auto route to the proper service. I used it to escalate different groups of servers to different teams based on a lookup

https://github.com/davpx/splunk_app_pagerduty

0 Karma

pfernandez133
Explorer

Well, it's not the most elegant solution but...I created an additional Splunk service in PagerDuty, called Splunk-X. Then, on the Splunk server, I copied ~splunk/scripts/pagerduty.py to ~splunk/scripts/pagerduty-x.py. I then changed the following line in def main():

pagerduty_api_key = get_pagerduty_api_key(config_file)

to the following line:

pagerduty_api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Where xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx is the service API from Splunk-X in PagerDuty. Then, I followed PagerDuty's steps on creating an alert. Namely, you create an alert, then set it to run a script, and that script is pagerduty-x.py.

Tested. Works. Yay!

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...