This app is sending over an incident_key parameter in each message when triggering a PagerDuty alert. The incident_key parameter seems to default to the Splunk saved search name.
This is causing an issue: when an alert triggers multiple times, the message is attributed to the already open PagerDuty incident (which had been acknowledged) and not triggering a new page.
I need to be able to update the incident_key parameter to either not use the Splunk saved search name and instead use a token from the search results, or just do something basic like throw a datestamp in there, so that a new pager duty incident is created each time the Splunk alert fires.
I've poked around in the app code and the area where the saved search name is attributed to the incident_key isn't jumping out at me, so looking to see if anyone else has successfully changed this parameter or has any suggestions.
Example of the json snippet that PD app is sending to the PD events API:
{
"client": "Splunk",
"client_url": ".../app/SplunkEnterpriseSecuritySuite/@go?sid=scheduler__xxx__xxx_at_1514723580_78469",
"description": "Saved Search Name",
"details": {
"info": "high",
"column": "severity"
},
"event_type": "trigger",
"incident_key": "Saved Search Name",
"service_key": "xxxxxxxxxxxx"
}
... View more