All Apps and Add-ons

WebTools app - issues with POST bodies under version 1.2.6

runner724
Path Finder

Using version 1.2.6 of WebTools, I am running into into an issue where POSTs with request bodies are being rejected by the service I am calling due to invalid/mutilated JSON. When I run the exact same queries against WebTools version 1.2.2, they work fine. I see some subtle differences in the app's bin\curl.py file that I think are responsible.

Here is the high-level format of my query:

| makeresults 
| eval header="{\"Content-Type\":\"application/json\"}" 
| eval payload="(...)" 
| curl method=post ssl=true user=(...) pass=(...) uri="(...)" datafield=payload headerfield=header debug=true
(...)

I include the "debug=true" to demonstrate a slight difference between 1.2.2 and 1.2.6/.

Under 1.2.6, a payload like this...

{ "query": { "bool": { "must": [ {"term": (...)
... results in the 'curl_data_payload' output of:
{ "query": { "bool": { "must": [ {"term": (...)
However, under 1.2.6, the same payload results in the 'curl_data_payload' output of:
{ u'query': { u'bool': { u'must': [ {u'term': (...)
which makes sense when I look at the bin\curl.py file and find this line:
data = json.loads(result[options['datafield']])

I played around with bin\curl.py until I could get things to work. If I take this text:

# STREAMING Use Case: iterate through results and run curl commands
if len(results) > 0:
    for result in results:
        # use JSON encoded header string if provided
        if 'headerfield' in options:
            headers = json.loads(result[options['headerfield']])
        else:
            headers = None

        # if data in options, set data = options['data']
        if 'data' in options:
            data = str(options['data'])

        # if datafield in options, set datafield = options['datafield']
        if 'datafield' in options:
            try:
                data = json.loads(result[options['datafield']])
            except:
                data = str(result[options['datafield']])
        else:
            data = None

        # debugging option
        if 'debug' in options:
            if options['debug'].lower() in ("yes", "true", "t", "1"):
                # for debugging we add results which show the options \
                # that were sent to the curl command
                result['curl_method'] = method
                result['curl_verifyssl'] = verifyssl
                result['curl_uri'] = uri
                result['curl_splunkauth'] = splunkauth
                if data != None:
                    result['curl_data_payload'] = data
                if headers:
                    result['curl_header'] = headers

        # based on method, execute appropriate function
        if method.lower() in ("get","g"):
            Result = get(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("head","h"):
            Result = head(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("post","p"):
            Result = post(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("put"):
            Result = put(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("delete","del","d"):
            Result = delete(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)

... and make a few adjustments:

# STREAMING Use Case: iterate through results and run curl commands
if len(results) > 0:
    for result in results:
        # use JSON encoded header string if provided
        if 'headerfield' in options:
            headers = json.loads(result[options['headerfield']])
        else:
            headers = None

        # if data in options, set data = options['data']
        if 'data' in options:
            data = str(options['data'])

        # if datafield in options, set datafield = options['datafield']
        if 'datafield' in options:
            data = str(options['datafield'])
        else:
            data = None

        # debugging option
        if 'debug' in options:
            if options['debug'].lower() in ("yes", "true", "t", "1"):
                # for debugging we add results which show the options \
                # that were sent to the curl command
                result['curl_method'] = method
                result['curl_verifyssl'] = verifyssl
                result['curl_uri'] = uri
                result['curl_splunkauth'] = splunkauth
                if data != None:
                    result['curl_data_payload'] = result[data]
                if headers:
                    result['curl_header'] = headers

        # based on method, execute appropriate function
        if method.lower() in ("get","g"):
            Result = get(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("head","h"):
            Result = head(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("post","p"):
            Result = post(uri,sessionKey,verifyssl,headers,result[data],user,passwd,timeout)
        if method.lower() in ("put"):
            Result = put(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("delete","del","d"):
            Result = delete(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)

The queries will work once I make those adjustments.

To be honest, I don't know enough about Python to explain why my adjustments fix things. I basically just looked at how 1.2.2 did things and adjusted the 1.2.6 to match.

Is anybody else running into cases where POSTs with a request body are rejected after updating WebTools to 1.2.6?

Splunk version: 7.1.1
'requests' python module version: 2.6.0
python version: 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

1 Solution

jkat54
SplunkTrust
SplunkTrust

Someone suggested we make it json a while back.

You could remove the json.loads(...) from the data = lines and it should revert the behavior.

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Someone suggested we make it json a while back.

You could remove the json.loads(...) from the data = lines and it should revert the behavior.

runner724
Path Finder

You're right, that is sufficient to fix the POST issues that I have, and is cleaner than changing three lines.

data = json.loads(result[options['datafield']])
->
data = str(result[options['datafield']])

Thanks!

jkat54
SplunkTrust
SplunkTrust

I converted to answer, please let us know if it was acceptable by accepting the answer or not. Cheers!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...