topic Re: PagerDuty incomplete results received in All Apps and Add-ons

PagerDuty incomplete results received

cdstealer — Thu, 23 Feb 2017 11:30:24 GMT

Hi,
We use PagerDuty for security results for the oncall security personnel. Unfortunately the alerts received by PagerDuty are incomplete. Only the first event is being received. Also, if there are multiple results, the results in pagerduty are also wrong. The first splunk result returned 3 usernames, pagerduty says 5 and it contains duplicates.

The splunk server is running 6.3.0 (we are unable to upgrade it currently) and the pagerduty app, both version 1.0 & 1.1 do the same.

I have attached both the splunk results and the pageduty received results. Unfortunately nearly all details needed to be obscured, but hopefully they will show what I mean.

Thanks
Steve

Re: PagerDuty incomplete results received

woodcock — Thu, 23 Feb 2017 16:43:00 GMT

You will need to explain the exact method and configuration file details on how you are getting those events into Splunk.

Re: PagerDuty incomplete results received

cdstealer — Mon, 27 Feb 2017 07:21:38 GMT

Hi Woodcock,
It's just a scheduled search that looks for specific patterns in the security device logs and if there are any matches, it sends the results to the pager duty app and email recipients. Unfortunately I'm very limited in the info I can provide as this is customer data but if there is anything specific you need to know, then I'll provide what I can. The only thing we add to the pagerduty app is the API key.

Thanks

Re: PagerDuty incomplete results received

woodcock — Tue, 28 Feb 2017 20:43:27 GMT

I am talking about the details regarding "it sends results to pagerduty". Obviously something is misconfigured there but you have given us absolutely no details there. You didn't even show us the alert configuration from savedsearches.conf, which might give us some idea of how that part works.

Re: PagerDuty incomplete results received

cdstealer — Thu, 02 Mar 2017 09:17:57 GMT

Hi Woodcock, Here is a savedsearch that we've used.

    [00_pagerduty_test]
    action.email.include.results_link = 0
    action.email.inline = 1
    action.email.message.alert = Multiple Failed please investigate
    action.email.sendcsv = 1
    action.email.sendresults = 1
    action.pagerduty = 1
    alert.severity = 4
    alert.suppress = 0
    alert.track = 1
    auto_summarize.dispatch.earliest_time = -1d@h
    counttype = number of events
    cron_schedule = * * * * *
    dispatch.earliest_time = -60m@m
    dispatch.latest_time = @m
    enableSched = 1
    quantity = 0
    relation = greater than
    search = index=f5 attack_type="*Other application activity*" response_code=200 username!="XXX*" (ip_client!="xxx.xxx.xxx.xxx" OR ip_client="xxx.xxx.xxx.xxx" OR ip_client!="xxx.xxx.xxx.xxx/19" OR ip_client!="xxx.xxx.xxx.xxx/20" OR ip_client="xxx.xxx.xxx.xxx" OR ip_client="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/19" OR ip_client!="xxx.xxx.xxx.xxx/18" OR ip_client!="xxx.xxx.xxx.xxx/16" OR ip_client!="xxx.xxx.xxx.xxx/17" OR ip_client="xxx.xxx.xxx.xxx/19" OR ip_client!="xxx.xxx.xxx.xxx/26" OR ip_client!="xxx.xxx.xxx.xxx" OR ip_client!="xxx.xxx.xxx.xxx" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24" OR ip_client!="xxx.xxx.xxx.xxx/24") sig_names!="*XXXXXXXXXX*" | stats  dc(username) AS distinctUsers values(username) by ip_client, uri  | where  distinctUsers > 20
disabled = 0

As I mentioned previously, the only config we have done for the PagerDuty app is to add the API key that gets inserted into the URL it uses to send the results to, so I'm unsure as to what we possibly could have misconfigured. But I have an open mind 🙂

Thanks

Re: PagerDuty incomplete results received

woodcock — Thu, 02 Mar 2017 18:57:16 GMT

If that is truly all the configuration that there is, I don't see any way to proceed other than to get Pagerduty involved.

Re: PagerDuty incomplete results received

cdstealer — Mon, 06 Mar 2017 06:49:23 GMT

agreed.. 🙂 Thank you for you time, it's appreciated. Once I have this resolved, I'll update with the answer.

Re: PagerDuty incomplete results received

gdsahine — Wed, 27 Feb 2019 16:19:36 GMT

Run into this issue myself, by default the Pagerduty integration for Splunk deduplicates events on Search.

In PagerDuty -:

Services → Integrations → Integrations for Splunk → Edit integration

Under 'deduplicate on' options list -:

Search <- default option
Component
Host
Source
If open incident - attach results to it
Don’t deduplicate

If you select 'Don't deduplicate' you should find Pagerduty generate an incident for each event (if desired).

Re: PagerDuty incomplete results received

woodcock — Fri, 01 Mar 2019 06:08:53 GMT

Make sure that you report back here what the final resolution is when PagerDuty gets you the answer.

Re: PagerDuty incomplete results received

rmyerspin — Tue, 27 Aug 2019 20:46:51 GMT

This is EXACTLY the answer I was looking for. Thank you!