Splunk Cloud Platform

How to configure PagerDuty alerts in Splunk Cloud?

TechVeera
Engager

I've run into a few different issues with the Pager Duty integration in Splunk Cloud.

The documentation on Pager Duty's site is either outdated, not applicable to Splunk Cloud or else there's something wrong with the way my Splunk Cloud account is configured (could be a permissions issue): https://www.pagerduty.com/docs/guides/splunk-integration-guide/. I don't see an Alert Actions page in Splunk Cloud, I have a Searches, Reports and Alerts page though.

I've configured PD alerts in Splunk using the alertlogevent app but it's not clear if I should instead be using some other app. These alerts do fire when there are search hits but I'm seeing another issue (below). The alert web hook app type seems like it might be appropriate but I was unable to get it to work correctly. I cannot create an alert type using the pager dutyi ncident app. . . although I can set it as a Trigger Action (I guess this is how it's supposed to work, I don't find the UI to intuitive here).

When my alerts fire and create incidents in Pager Duty, I do not see a way to set the Pager Duty incident severity.

Also, the PD incidents include a link back to Splunk, which I believe should open the query with the search hits which generated the alert. However, the link brings me to a page with a Page Not Found! error. It contains a link to "more information about my request" which brings up a Splunk query with no hits. This query looks like "index=_internal, host=SOME_HOST_ON_SPLUNK_CLOUD, source=*web_service.log, log_level=ERROR, requestid=A_REQUEST_ID". It it not clear to me if this is a config issue, bug in Splunk Cloud or possibly even a permissions issue for my account.

Any help is appreciated.

Labels (2)
0 Karma

yourDevSre
New Member

FYI: I am using the the Splunk integration key to send the alert from SPLUNK to PagerDuty
For the incident priority, You can to create a new field in you Splunk search called severity which classifies the priority of the alert , its values determine the priority of the incident

critical P1
errorP2
warningP3
infoP4

Pager Duty is intelligent enough to parse the JSON payload 

Example: [splunk search...] | eval severity="critical" 

0 Karma
Get Updates on the Splunk Community!

Monitoring MariaDB and MySQL

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to ...

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...