PagerDuty incomplete results received

[cdstealer]

Hi,
We use PagerDuty for security results for the oncall security personnel. Unfortunately the alerts received by PagerDuty are incomplete. Only the first event is being received. Also, if there are multiple results, the results in pagerduty are also wrong. The first splunk result returned 3 usernames, pagerduty says 5 and it contains duplicates.

The splunk server is running 6.3.0 (we are unable to upgrade it currently) and the pagerduty app, both version 1.0 & 1.1 do the same.

I have attached both the splunk results and the pageduty received results. Unfortunately nearly all details needed to be obscured, but hopefully they will show what I mean.

Thanks
Steve

[gdsahine]

Run into this issue myself, by default the Pagerduty integration for Splunk deduplicates events on Search.

In PagerDuty -:

Services → Integrations → Integrations for Splunk → Edit integration

Under 'deduplicate on' options list -:

• Search <- default option
• Component
• Host
• Source
• If open incident - attach results to it
• Don’t deduplicate

If you select 'Don't deduplicate' you should find Pagerduty generate an incident for each event (if desired).

[rmyerspin]

This is EXACTLY the answer I was looking for. Thank you!

[woodcock]

If that is truly all the configuration that there is, I don't see any way to proceed other than to get Pagerduty involved.

[cdstealer]

agreed.. 🙂 Thank you for you time, it's appreciated. Once I have this resolved, I'll update with the answer.

[woodcock]

Make sure that you report back here what the final resolution is when PagerDuty gets you the answer.

[woodcock]

You will need to explain the exact method and configuration file details on how you are getting those events into Splunk.

[cdstealer]

Hi Woodcock,
It's just a scheduled search that looks for specific patterns in the security device logs and if there are any matches, it sends the results to the pager duty app and email recipients. Unfortunately I'm very limited in the info I can provide as this is customer data but if there is anything specific you need to know, then I'll provide what I can. The only thing we add to the pagerduty app is the API key.

Thanks

[woodcock]

I am talking about the details regarding "it sends results to pagerduty". Obviously something is misconfigured there but you have given us absolutely no details there. You didn't even show us the alert configuration from savedsearches.conf, which might give us some idea of how that part works.

[cdstealer]

Hi Woodcock, Here is a savedsearch that we've used.

    [00_pagerduty_test]
    action.email.include.results_link = 0
    action.email.inline = 1
    action.email.message.alert = Multiple Failed please investigate
    action.email.sendcsv = 1
    action.email.sendresults = 1
    action.pagerduty = 1
    alert.severity = 4
    alert.suppress = 0
    alert.track = 1
    auto_summarize.dispatch.earliest_time = -1d@h
    counttype = number of events
    cron_schedule = * * * * *
    dispatch.earliest_time = -60m@m
    dispatch.latest_time = @m
    enableSched = 1
    quantity = 0
    relation = greater than
    search = index=f5 attack_type="*Other application activity*" response_code=200 username!="XXX*" (ip_client!="xxx.xxx.xxx.xxx" OR ip_client="xxx.xxx.xxx.xxx" OR ip_client!="xxx.xxx.xxx.xxx/19" OR ip_client!="xxx.xxx.xxx.xxx/20" OR ip_client="xxx.xxx.xxx.xxx" OR ip_client="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/19" OR ip_client!="xxx.xxx.xxx.xxx/18" OR ip_client!="xxx.xxx.xxx.xxx/16" OR ip_client!="xxx.xxx.xxx.xxx/17" OR ip_client="xxx.xxx.xxx.xxx/19" OR ip_client!="xxx.xxx.xxx.xxx/26" OR ip_client!="xxx.xxx.xxx.xxx" OR ip_client!="xxx.xxx.xxx.xxx" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24") sig_names!="*XXXXXXXXXX*" | stats  dc(username) AS distinctUsers values(username) by ip_client, uri  | where  distinctUsers > 20
disabled = 0

As I mentioned previously, the only config we have done for the PagerDuty app is to add the API key that gets inserted into the URL it uses to send the results to, so I'm unsure as to what we possibly could have misconfigured. But I have an open mind 🙂

Thanks