Problem with TA-Webtools curl POST method with JSON datafield?

[bowesmana]

I think there may be a bug with the use of datafield in the POST operation with curl. Currently the curl.py python (line 259) code does

data = json.loads(result[options['datafield']])

which will create a Python object called ‘data’ containing the parsed JSON fragment. In my example, my JSON payload is

{"devices": ["arn:aws:sns:ap-southeast-2:000000000000:endpoint/GCM/bcrm/cabf12dc-ec12-3af4-12e1-121a193ecc7b"], "message": {"title": "Reminder to complete your self checkup", "body": "Your last completed self checkup is to old. Please perform the self checkup before attending the workplace"}}

The curl_data_payload is then output with the debug=true setting as

{u'devices': [u'arn:aws:sns:ap-southeast-2:000000000000:endpoint/GCM/bcrm/cabf12dc-ec12-3af4-12e1-121a193ecc7b'], u'message': {u'title': u'Reminder to complete your self checkup', u'body': u'Your last completed self checkup is to old. Please perform the self checkup before attending the workplace'}}

When this gets passed into the requests.post call, it appears to then send the string representation of the Python object rather than the string data itself. The receiving system fails to parse the received object as it is not JSON any more. If I change the code to be

data = str(result[options['datafield']])

then this will send OK as the curl_data_payload is then

{"devices": ["arn:aws:sns:ap-southeast-2: 000000000000:endpoint/GCM/bcrm/cabf12dc-ec12-3af4-12e1-121a193ecc7b"], "message": {"title": "Reminder to complete your self checkup", "body": "Your last completed self checkup is to old. Please perform the self checkup before attending the workplace"}}

I am not sure what the intention of the datafield option is if it always tried to JSON parse the data with json.loads() and it seems to me that it will always fail if the data is valid JSON.

Is this a bug or just a problem with how I am using curl?

[jkat54]

Would you like to submit a fix on github?

No finders fees for the bug reporting but lots of fame and notoriety!

https://github.com/bentleymi/Splunk/tree/master/TA-webtools

[bowesmana]

Thanks for the update.

Happy to submit a fix - but was really looking to understand if what I deem a fix is actually correct under the original implementation, as it looks like you've taken pains to do the json.loads and catch exceptions and I don't get why.

[jkat54]

Got it!

well as the story goes, once upon a time someone came and learned us all why it should be json.loads(), etc, and I released a patch where it would be... looks like I missed some code though.

so afaik its a bug you've found and everything should be json with the "u"s.

[bowesmana]

@jkat54 

I raised a ticket on github

https://github.com/bentleymi/ta-webtools/issues/15

there's not an obvious solution, but I've put a hack solution in the ticket which will work for me for now.