Archive
Highlighted

TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

Explorer

Expected behaviour

TA-Webtools, when posting a JSON body, posts successfully.

Actual behaviour & Potential PBCAK

When posting a JSON body to the data field via the curl command in Splunk, the JSON body is truncated. While a successful post is made, the JSON body that is presented to Splunk's REST API is only the first characters before the first JSON internal double quote. I did attempt to use both single, double, and triple slashes to escape the quotes, all which result in malformed eval commands.

For example:

"{\"entity_rules\": [{\"rule_items\":

Presents the Splunk API with the payload:

{

SPL Context

This is the end of the command where we are assembling the string that will be used by the curl command

| eval curl_command=curl_command+" "+"data=" +"{\"entity_rules\": [{\"rule_items\": [{\"field_type\": \"info\", \"field\": \"parentserviceinfo\", \"rule_type\": \"matches\", \"value\": \"deletemeparentservice-dc100\"}], \"rule_condition\": \"AND\"}], \"permissions\": {\"read\": true, \"group\": {\"read\": true, \"delete\": true, \"write\": true}, \"user\": \"admin\", \"delete\": true, \"write\": true}, \"object_type\": \"service\", \"sec_grp\": \"default_itsi_security_group\"}"
| map search="| curl method=post uri=$curl_command$ user=admin pass=OMITTED debug=t"

Steps to reproduce

  1. Install latest TA-webtools distribution
  2. Open the TA-webtools permission to allow use of curl to other apps
  3. Switch to ITSI app
  4. Verify you can use curl within ITSI search
  5. Use the curl command, with post argument, and pass a JSON body to the data curl parameter that contains no quotes

Proposed Fix

While this is not definite, it has been suggested in the Requests Library Documentation that you can encode the payload as JSON using the JSON library's function:
json.dump(param)

While this may not be the complete fix, I think it may be worth time investigating.

Screenshots

Documentation requests lib
Splunk search results
Desired Changes in curl.py
Previewing curl command before executing

Configuration

  • TA-Webtools Version: Version 1.30
  • Splunk version: Splunk 7.0.2 (build 03bbabbd5c0f)
  • In context of Splunk App and version: ITSI, 3.01
  • OS: Centos 7.4 x86_64
  • Browser: chrome
Highlighted

Re: TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

Path Finder

Can you post version of CLI curl that successfully updated ITSI entity rule? That way we will know what worked. Thanks.

0 Karma
Highlighted

Re: TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

Explorer

This is the working command via bash
https://pastebin.com/WhkMNj2G

0 Karma
Highlighted

Re: TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

SplunkTrust
SplunkTrust

The curl SPL command supports streaming data through as well, have you tried this approach? Here's an example:

 | makeresults count=1 | eval message="{'name':'restart_link','value':'Hello World','severity':'warn'}" | curl method=post uri=https://localhost:8089/services/messages/new datafield=message splunkauth=true | table curl*

You might also try swapping single for double quotes as in the above example

Since you may post JSON and someone else might post XML, the command is designed to take whatever data payload or data field in the pipeline and get/post/delete with it.

I cant just arbitrarily assume you'll be sending JSON and therefore json.dumps(payload).

If you want to urlencode, the app ships with a urlencode command as well.

It can be tricky getting the command to work but the pay off is good too 🙂

Try adding debug=t to the command as well... it can help to see what it receives as inputs.

0 Karma
Highlighted

Re: TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

Path Finder

We need to do change the uri as well which is why we went to map. Any tips on how to sub a field into the uri argument welcome.

0 Karma
Highlighted

Re: TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

SplunkTrust
SplunkTrust

We did it in this example after someone came up with the use case:

Deleting fired alerts for search named "Test Alert":
| rest /servicesNS/admin/search/alerts/fired_alerts/Test%20Alert | fields title | head 10 | map search="|curl method=delete uri="https://localhost:8089/servicesNS/admin/search/alerts/fired_alerts/$title$" user=admin pass=changeme | table *"

As you can see, we take the random search ids (title) from the splunk rest, and then map it to the URI used in the curl command in order to delete triggered alerts via the curl command.

0 Karma
Highlighted

Re: TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

SplunkTrust
SplunkTrust

i see you're getting really close in one of your screenshots

0 Karma
Highlighted

Re: TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

SplunkTrust
SplunkTrust

instead of making one field that maps through try more than one...

one for URI and another for data

| map search="|curl uri=$URI$ data=$PAYLOAD$" for example

Highlighted

Re: TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

Path Finder

Good idea.

0 Karma
Highlighted

Re: TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

Explorer

Thanks for the quick reply, and support on this matter.

I tried using two variables to map this instead of one concatenated variable and I wasn't able to find success with that either.
Its also worth noting I tried using single quotes in the JSON to no avail.

Current_status

0 Karma