Alerting

Has anyone experienced cases where Webhook alert payloads are including strange 'loadjob' calls in the result_link??

paimonsoror
Builder

I recently got alerted by one of my customers that they received some strange results from one of their webhook payloads. For some reason, the payload contains a TON of 'AD' related information (we use LDAP auth for Splunk). Luckily no sensitive data is exposed, but the whole thing is incredibly bizarre.

I am used to seeing a webhook payload like this:

{
    "app": "my_super_fun_app", 
    "sid": "scheduler__admin_Y2lnbmFfY2hhcmdlYmFjaw__RMD512f1091d029f2c3a_at_1509948000_79253", 
    "owner": "admin", 
    "search_name": "Usage Drift Alert", 
    "result": {
        "Primary Contact": "--", 
        "Total Used GB": "111.54", 
        "Quota Percentage": "56.53",
        "Usage Drift": "47.86", 
        "Index": "app_sharepoint"
    }, 
    "results_link": "http://obfuscated_this:8000/app/my_super_fun_app/@go?sid=scheduler__admin_Y2lnbmFfY2hhcmdlYmFjaw__RMD512f1091d029f2c3a_at_1509948000_79253"
}

Take a look at the results_link above, it follows the typical pattern of '@go?sid=[search_sid]"

However, my customer got a result like the following:

{
    "results_link": "https://OBFUSCATED.com/app/app_harmony_pvs/search?q=%7Cloadjob%20scheduler__m38437_YXBwX2hhcm1vbnlfcHZz__RMD529b0cadb7d414a12_at_1509996600_238_D644C3A5-C833-4F31-8A77-5B88CA230AB1%20%7C%20head%2011%20%7C%20tail%201&earliest=0&latest=now",
    "app": "app_harmony_pvs",
    "search_name": "Harmony Timeouts",
    "owner": "m38437",
    "result": {
        "results.n36014.attributes.codePage{}": "",
        "results.c77827.attributes.sIDHistory{}.encoding": "",
        "results.mvruss.dn": "",
        "results.c30258.attributes.objectSid{}": "",
        "results.c30789.attributes.postalCode{}": "",
        "results.c30789.attributes.msExchHideFromAddressLists{}": "",
        "results.n36014.attributes.thumbnailPhoto{}.encoding": "",
        "results.c77827.attributes.managedObjects{}": "",
        "enabled": "",
        "app": "",
        "source": "/usr/local/openresty/nginx/logs/error.log",
        "results.spshep.attributes.homeMDB{}": "",
        "status": "",
        "returnappsonly": "false",
        "results.c77827.attributes.homeDrive{}": "",
        "results.c24086.dn": "",
        "date_wday": "monday",
        "application_server": "",
        "splunk_server": "cMASKED0010",
        "G": "",
        "range": "",
        "change_type": "",
        "ignorechanges": "false",
        "vendor_product": "",
        "timestamp": "",
        "vendor": "",
        "_sourcetype": "harmony:openresty:error",
        "date_zone": "local",
        "date_year": "2017",
        "DC": "",
        "_eventtype_color": "none",
        "Endpoint": "/CI/Relationship/iquote",
        "status_type": "",
        "results.cmedwa.attributes.manager{}": "",
        "results.c77827.attributes.msExchVersion{}": "",
        "_raw": "OBFUSCATED",
        "results.c30258.attributes.initials{}": "",
        "eventtype": [
            "err0r",
            "nix-all-logs",
            "nix_errors"
        ],
        "_indextime": "1509995250",
        "start_time": "",
        "status_description": "",
        "tag::app": "",
        "timestartpos": "0",
        "P": "",
        "punct": "//_::_[]_#:_*____(:___)______,_:_...,_:_,_:_\"_////",
        "_time": "1509995250",
        "_kv": "1",
        "user_type": "",
        "linecount": "1",
        "meta": "",
        "O": "",
        "timeendpos": "20",
        "unix_category": "all_hosts",
        "unix_group": "default",
        "CN": "",
        "tag": "error",
        "_cd": "106:4934192",
        "Verb": "GET",
        "object_category": "",
        "A": "",
        "tag::eventtype": "error",
        "date_hour": "14",
        "_bkt": "app_harmony~106~26CC9C87-0F68-4672-B76C-6C10DF00A4E2",
        "host": "cMASKED0047",
        "date_mday": "6",
        "o": "",
        "_serial": "1",
        "date_month": "november",
        "product": "",
        "src": "",
        "appserver_port_number": "",
        "splunk_server_group": "",
        "index": "app_harmony"
    },
    "sid": "scheduler__m38437_YXBwX2hhcm1vbnlfcHZz__RMD529b0cadb7d414a12_at_1509996600_238_D644C3A5-C833-4F31-8A77-5B88CA230AB1"
}

The bizaare thing is that the actual result set was a total of 12 results, with only a handful of extractions, however, if you focus on the 'result.results' node, it is a bunch of stuff from AD. I think it is only listing the users that are in the groups of the customer, and I removed a ton of them before posting, but hopefully you get the picture.

Also, take a look at the "results_link". it is completely different than the common sid based search, and instead is using a 'q=' with a loadjob. And then some stuff with a tail and 'head'.

I cant for the life of me figure out what is going on here.

0 Karma

damien_chillet
Builder

Regarding formatting of the result_link:

From my experience with Splunk talking to Jira, I could observe that
loadjob is used with head and tail when an alert is used in 'per result mode'.

If your search returned 3 results for example, there will be 3 alerts and the link for each will be:

head 1 tail 1 for the first
head 2 tail 1 for the second
head 3 tail 1 for the third

If the alert is configured in digest mode (that means you get one alert with all results provided by the search inside), Splunk will provide a link with "@go?sid="

0 Karma

jkat54
SplunkTrust
SplunkTrust

What search are you using to see this data?

0 Karma

paimonsoror
Builder

The search is:

index=app_harmony "*timed out*"

And set for -1h to now
Alert once per result

The strangest thing about the whole thing, is that this was working fine earlier today. Nothing changed on the search, and nothing changed in the infrastructure.

0 Karma

jkat54
SplunkTrust
SplunkTrust

One result says index is app harmony the other says index is app share point. Is someone feeding data into the wrong index by way of a bad inputs.conf? Maybe the stanza name is malformed

0 Karma

paimonsoror
Builder

The first result is a different search. I was using that one to demonstrate what we normally see for the formatting of the results_link .

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Sorry but I misunderstood question earlier so I removed my comment

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...