All Apps and Add-ons

PagerDuty incomplete results received

cdstealer
Contributor

Hi,
We use PagerDuty for security results for the oncall security personnel. Unfortunately the alerts received by PagerDuty are incomplete. Only the first event is being received. Also, if there are multiple results, the results in pagerduty are also wrong. The first splunk result returned 3 usernames, pagerduty says 5 and it contains duplicates.

The splunk server is running 6.3.0 (we are unable to upgrade it currently) and the pagerduty app, both version 1.0 & 1.1 do the same.

I have attached both the splunk results and the pageduty received results. Unfortunately nearly all details needed to be obscured, but hopefully they will show what I mean.

splunk results
pagerduty results

Thanks
Steve

0 Karma

gdsahine
Engager

Run into this issue myself, by default the Pagerduty integration for Splunk deduplicates events on Search.

In PagerDuty -:

Services → Integrations → Integrations for Splunk → Edit integration

Under 'deduplicate on' options list -:

  • Search <- default option
  • Component
  • Host
  • Source
  • If open incident - attach results to it
  • Don’t deduplicate

If you select 'Don't deduplicate' you should find Pagerduty generate an incident for each event (if desired).

rmyerspin
Engager

This is EXACTLY the answer I was looking for. Thank you!

0 Karma

woodcock
Esteemed Legend

If that is truly all the configuration that there is, I don't see any way to proceed other than to get Pagerduty involved.

0 Karma

cdstealer
Contributor

agreed.. 🙂 Thank you for you time, it's appreciated. Once I have this resolved, I'll update with the answer.

0 Karma

woodcock
Esteemed Legend

Make sure that you report back here what the final resolution is when PagerDuty gets you the answer.

0 Karma

woodcock
Esteemed Legend

You will need to explain the exact method and configuration file details on how you are getting those events into Splunk.

0 Karma

cdstealer
Contributor

Hi Woodcock,
It's just a scheduled search that looks for specific patterns in the security device logs and if there are any matches, it sends the results to the pager duty app and email recipients. Unfortunately I'm very limited in the info I can provide as this is customer data but if there is anything specific you need to know, then I'll provide what I can. The only thing we add to the pagerduty app is the API key.

Thanks

0 Karma

woodcock
Esteemed Legend

I am talking about the details regarding "it sends results to pagerduty". Obviously something is misconfigured there but you have given us absolutely no details there. You didn't even show us the alert configuration from savedsearches.conf, which might give us some idea of how that part works.

0 Karma

cdstealer
Contributor

Hi Woodcock, Here is a savedsearch that we've used.

    [00_pagerduty_test]
    action.email.include.results_link = 0
    action.email.inline = 1
    action.email.message.alert = Multiple Failed please investigate
    action.email.sendcsv = 1
    action.email.sendresults = 1
    action.pagerduty = 1
    alert.severity = 4
    alert.suppress = 0
    alert.track = 1
    auto_summarize.dispatch.earliest_time = -1d@h
    counttype = number of events
    cron_schedule = * * * * *
    dispatch.earliest_time = -60m@m
    dispatch.latest_time = @m
    enableSched = 1
    quantity = 0
    relation = greater than
    search = index=f5 attack_type="*Other application activity*" response_code=200 username!="XXX*" (ip_client!="xxx.xxx.xxx.xxx" OR ip_client="xxx.xxx.xxx.xxx" OR ip_client!="xxx.xxx.xxx.xxx/19" OR ip_client!="xxx.xxx.xxx.xxx/20" OR ip_client="xxx.xxx.xxx.xxx" OR ip_client="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/19" OR ip_client!="xxx.xxx.xxx.xxx/18" OR ip_client!="xxx.xxx.xxx.xxx/16" OR ip_client!="xxx.xxx.xxx.xxx/17" OR ip_client="xxx.xxx.xxx.xxx/19" OR ip_client!="xxx.xxx.xxx.xxx/26" OR ip_client!="xxx.xxx.xxx.xxx" OR ip_client!="xxx.xxx.xxx.xxx" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24") sig_names!="*XXXXXXXXXX*" | stats  dc(username) AS distinctUsers values(username) by ip_client, uri  | where  distinctUsers > 20
disabled = 0

As I mentioned previously, the only config we have done for the PagerDuty app is to add the API key that gets inserted into the URL it uses to send the results to, so I'm unsure as to what we possibly could have misconfigured. But I have an open mind 🙂

Thanks

0 Karma
Get Updates on the Splunk Community!

Discover Powerful New Features in Splunk Cloud Platform: Enhanced Analytics, ...

Hey Splunky people! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2408. In this ...

Splunk Classroom Chronicles: Training Tales and Testimonials

Welcome to the "Splunk Classroom Chronicles" series, created to help curious, career-minded learners get ...

Access Tokens Page - New & Improved

Splunk Observability Cloud recently launched an improved design for the access tokens page for better ...