TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

[phastings]

Expected behaviour

TA-Webtools, when posting a JSON body, posts successfully.

Actual behaviour & Potential PBCAK

When posting a JSON body to the data field via the curl command in Splunk, the JSON body is truncated. While a successful post is made, the JSON body that is presented to Splunk's REST API is only the first characters before the first JSON internal double quote. I did attempt to use both single, double, and triple slashes to escape the quotes, all which result in malformed eval commands.

For example:

"{\"entity_rules\": [{\"rule_items\":

Presents the Splunk API with the payload:

{


SPL Context

This is the end of the command where we are assembling the string that will be used by the curl command

| eval curl_command=curl_command+" "+"data=" +"{\"entity_rules\": [{\"rule_items\": [{\"field_type\": \"info\", \"field\": \"parentserviceinfo\", \"rule_type\": \"matches\", \"value\": \"deletemeparentservice-dc100\"}], \"rule_condition\": \"AND\"}], \"permissions\": {\"read\": true, \"group\": {\"read\": true, \"delete\": true, \"write\": true}, \"user\": \"admin\", \"delete\": true, \"write\": true}, \"object_type\": \"service\", \"sec_grp\": \"default_itsi_security_group\"}"
| map search="| curl method=post uri=$curl_command$ user=admin pass=OMITTED debug=t"

Steps to reproduce

1. Install latest TA-webtools distribution
2. Open the TA-webtools permission to allow use of curl to other apps
3. Switch to ITSI app
4. Verify you can use curl within ITSI search
5. Use the curl command, with post argument, and pass a JSON body to the data curl parameter that contains no quotes

Proposed Fix

While this is not definite, it has been suggested in the Requests Library Documentation that you can encode the payload as JSON using the JSON library's function:
json.dump(param)

While this may not be the complete fix, I think it may be worth time investigating.

Screenshots

Configuration

• TA-Webtools Version: Version 1.30
• Splunk version: Splunk 7.0.2 (build 03bbabbd5c0f)
• In context of Splunk App and version: ITSI, 3.01
• OS: Centos 7.4 x86_64
• Browser: chrome

[jkat54]

The curl SPL command supports streaming data through as well, have you tried this approach? Here's an example:

 | makeresults count=1 | eval message="{'name':'restart_link','value':'Hello World','severity':'warn'}" | curl method=post uri=https://localhost:8089/services/messages/new datafield=message splunkauth=true | table curl*

You might also try swapping single for double quotes as in the above example

Since you may post JSON and someone else might post XML, the command is designed to take whatever data payload or data field in the pipeline and get/post/delete with it.

I cant just arbitrarily assume you'll be sending JSON and therefore json.dumps(payload).

If you want to urlencode, the app ships with a urlencode command as well.

It can be tricky getting the command to work but the pay off is good too 🙂

Try adding debug=t to the command as well... it can help to see what it receives as inputs.

[ian_thomas]

We need to do change the uri as well which is why we went to map. Any tips on how to sub a field into the uri argument welcome.

[jkat54]

We did it in this example after someone came up with the use case:

Deleting fired alerts for search named "Test Alert":
| rest /servicesNS/admin/search/alerts/fired_alerts/Test%20Alert | fields title | head 10 | map search="|curl method=delete uri="https://localhost:8089/servicesNS/admin/search/alerts/fired_alerts/$title$" user=admin pass=changeme | table *"

As you can see, we take the random search ids (title) from the splunk rest, and then map it to the URI used in the curl command in order to delete triggered alerts via the curl command.

[jkat54]

i see you're getting really close in one of your screenshots

[jkat54]

instead of making one field that maps through try more than one...

one for URI and another for data

| map search="|curl uri=$URI$ data=$PAYLOAD$" for example

[phastings]

Thanks for the quick reply, and support on this matter.

I tried using two variables to map this instead of one concatenated variable and I wasn't able to find success with that either.
Its also worth noting I tried using single quotes in the JSON to no avail.

[jkat54]

Can you paste the search somewhere that I can copy it out an try to replicate the same on my end?

[phastings]

Absolutely. Below is the current search. It is in the context of ITSI but the principal JSON as payload should be the same.

Thank you.

| rest splunk_server=local /servicesNS/nobody/SA-ITOA/itoa_interface/service fields="_key,entity_rules,title"
| eval strippedValue=spath(value,"{}")
| mvexpand strippedValue
`comment("breaking up JSON body into variables we can use to reusable fields")`
| eval service_id=spath(strippedValue,"_key"), service_title=spath(strippedValue,"title"), entity_rules_field=spath(strippedValue,"entity_rules{}"), newestValue=spath(strippedValue,"kpis{}")
`comment("return all results where the entity rules have not been applied")`
| search NOT entity_rules_field="*"
| eval comment3="adding service ID to act as a lookup key, to the URL"
| eval curl_command="https://localhost:8089/servicesNS/nobody/SA-ITOA/itoa_interface/service/".service_id
| eval comment4="adding partial data flag to the existing url. We want to only update the desired entity_rules today"
| eval curl_command=curl_command."/?is_partial_data=1"
`comment("adding data body flag with JSON body, there is a space to separate the URL from the data flag")`
| eval curl_command=curl_command
| eval dataInput="{\"entity_rules\": [{\"rule_items\": [{\"field_type\": \"info\", \"field\": \"parentserviceinfo\", \"rule_type\": \"matches\", \"value\": \"deletemeparentservice-dc100\"}], \"rule_condition\": \"AND\"}], \"permissions\": {\"read\": true, \"group\": {\"read\": true, \"delete\": true, \"write\": true}, \"user\": \"admin\", \"delete\": true, \"write\": true}, \"object_type\": \"service\", \"sec_grp\": \"default_itsi_security_group\"}"
| map search="| curl method=post uri=$curl_command$ data=$dataInput$ user=admin pass=omitted debug=t"

[ian_thomas]

Good idea.

[ian_thomas]

Can you post version of CLI curl that successfully updated ITSI entity rule? That way we will know what worked. Thanks.

[phastings]

This is the working command via bash
https://pastebin.com/WhkMNj2G