All Apps and Add-ons

WebTools app - issues with POST bodies under version 1.2.6

runner724
Path Finder

Using version 1.2.6 of WebTools, I am running into into an issue where POSTs with request bodies are being rejected by the service I am calling due to invalid/mutilated JSON. When I run the exact same queries against WebTools version 1.2.2, they work fine. I see some subtle differences in the app's bin\curl.py file that I think are responsible.

Here is the high-level format of my query:

| makeresults 
| eval header="{\"Content-Type\":\"application/json\"}" 
| eval payload="(...)" 
| curl method=post ssl=true user=(...) pass=(...) uri="(...)" datafield=payload headerfield=header debug=true
(...)

I include the "debug=true" to demonstrate a slight difference between 1.2.2 and 1.2.6/.

Under 1.2.6, a payload like this...

{ "query": { "bool": { "must": [ {"term": (...)
... results in the 'curl_data_payload' output of:
{ "query": { "bool": { "must": [ {"term": (...)
However, under 1.2.6, the same payload results in the 'curl_data_payload' output of:
{ u'query': { u'bool': { u'must': [ {u'term': (...)
which makes sense when I look at the bin\curl.py file and find this line:
data = json.loads(result[options['datafield']])

I played around with bin\curl.py until I could get things to work. If I take this text:

# STREAMING Use Case: iterate through results and run curl commands
if len(results) > 0:
    for result in results:
        # use JSON encoded header string if provided
        if 'headerfield' in options:
            headers = json.loads(result[options['headerfield']])
        else:
            headers = None

        # if data in options, set data = options['data']
        if 'data' in options:
            data = str(options['data'])

        # if datafield in options, set datafield = options['datafield']
        if 'datafield' in options:
            try:
                data = json.loads(result[options['datafield']])
            except:
                data = str(result[options['datafield']])
        else:
            data = None

        # debugging option
        if 'debug' in options:
            if options['debug'].lower() in ("yes", "true", "t", "1"):
                # for debugging we add results which show the options \
                # that were sent to the curl command
                result['curl_method'] = method
                result['curl_verifyssl'] = verifyssl
                result['curl_uri'] = uri
                result['curl_splunkauth'] = splunkauth
                if data != None:
                    result['curl_data_payload'] = data
                if headers:
                    result['curl_header'] = headers

        # based on method, execute appropriate function
        if method.lower() in ("get","g"):
            Result = get(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("head","h"):
            Result = head(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("post","p"):
            Result = post(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("put"):
            Result = put(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("delete","del","d"):
            Result = delete(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)

... and make a few adjustments:

# STREAMING Use Case: iterate through results and run curl commands
if len(results) > 0:
    for result in results:
        # use JSON encoded header string if provided
        if 'headerfield' in options:
            headers = json.loads(result[options['headerfield']])
        else:
            headers = None

        # if data in options, set data = options['data']
        if 'data' in options:
            data = str(options['data'])

        # if datafield in options, set datafield = options['datafield']
        if 'datafield' in options:
            data = str(options['datafield'])
        else:
            data = None

        # debugging option
        if 'debug' in options:
            if options['debug'].lower() in ("yes", "true", "t", "1"):
                # for debugging we add results which show the options \
                # that were sent to the curl command
                result['curl_method'] = method
                result['curl_verifyssl'] = verifyssl
                result['curl_uri'] = uri
                result['curl_splunkauth'] = splunkauth
                if data != None:
                    result['curl_data_payload'] = result[data]
                if headers:
                    result['curl_header'] = headers

        # based on method, execute appropriate function
        if method.lower() in ("get","g"):
            Result = get(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("head","h"):
            Result = head(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("post","p"):
            Result = post(uri,sessionKey,verifyssl,headers,result[data],user,passwd,timeout)
        if method.lower() in ("put"):
            Result = put(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("delete","del","d"):
            Result = delete(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)

The queries will work once I make those adjustments.

To be honest, I don't know enough about Python to explain why my adjustments fix things. I basically just looked at how 1.2.2 did things and adjusted the 1.2.6 to match.

Is anybody else running into cases where POSTs with a request body are rejected after updating WebTools to 1.2.6?

Splunk version: 7.1.1
'requests' python module version: 2.6.0
python version: 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

1 Solution

jkat54
SplunkTrust
SplunkTrust

Someone suggested we make it json a while back.

You could remove the json.loads(...) from the data = lines and it should revert the behavior.

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Someone suggested we make it json a while back.

You could remove the json.loads(...) from the data = lines and it should revert the behavior.

runner724
Path Finder

You're right, that is sufficient to fix the POST issues that I have, and is cleaner than changing three lines.

data = json.loads(result[options['datafield']])
->
data = str(result[options['datafield']])

Thanks!

jkat54
SplunkTrust
SplunkTrust

I converted to answer, please let us know if it was acceptable by accepting the answer or not. Cheers!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...