All Apps and Add-ons

TA-Webtools: Why is posting JSON via the curl command in TA-Webtools 1.3 is truncated?

phastings
Explorer

Expected behaviour

TA-Webtools, when posting a JSON body, posts successfully.

Actual behaviour & Potential PBCAK

When posting a JSON body to the data field via the curl command in Splunk, the JSON body is truncated. While a successful post is made, the JSON body that is presented to Splunk's REST API is only the first characters before the first JSON internal double quote. I did attempt to use both single, double, and triple slashes to escape the quotes, all which result in malformed eval commands.

For example:

"{\"entity_rules\": [{\"rule_items\":

Presents the Splunk API with the payload:

{

SPL Context

This is the end of the command where we are assembling the string that will be used by the curl command

| eval curl_command=curl_command+" "+"data=" +"{\"entity_rules\": [{\"rule_items\": [{\"field_type\": \"info\", \"field\": \"parentserviceinfo\", \"rule_type\": \"matches\", \"value\": \"deletemeparentservice-dc100\"}], \"rule_condition\": \"AND\"}], \"permissions\": {\"read\": true, \"group\": {\"read\": true, \"delete\": true, \"write\": true}, \"user\": \"admin\", \"delete\": true, \"write\": true}, \"object_type\": \"service\", \"sec_grp\": \"default_itsi_security_group\"}"
| map search="| curl method=post uri=$curl_command$ user=admin pass=OMITTED debug=t"

Steps to reproduce

  1. Install latest TA-webtools distribution
  2. Open the TA-webtools permission to allow use of curl to other apps
  3. Switch to ITSI app
  4. Verify you can use curl within ITSI search
  5. Use the curl command, with post argument, and pass a JSON body to the data curl parameter that contains no quotes

Proposed Fix

While this is not definite, it has been suggested in the Requests Library Documentation that you can encode the payload as JSON using the JSON library's function:
json.dump(param)

While this may not be the complete fix, I think it may be worth time investigating.

Screenshots

Documentation requests lib
Splunk search results
Desired Changes in curl.py
Previewing curl command before executing

Configuration

  • TA-Webtools Version: Version 1.30
  • Splunk version: Splunk 7.0.2 (build 03bbabbd5c0f)
  • In context of Splunk App and version: ITSI, 3.01
  • OS: Centos 7.4 x86_64
  • Browser: chrome

jkat54
SplunkTrust
SplunkTrust

The curl SPL command supports streaming data through as well, have you tried this approach? Here's an example:

 | makeresults count=1 | eval message="{'name':'restart_link','value':'Hello World','severity':'warn'}" | curl method=post uri=https://localhost:8089/services/messages/new datafield=message splunkauth=true | table curl*

You might also try swapping single for double quotes as in the above example

Since you may post JSON and someone else might post XML, the command is designed to take whatever data payload or data field in the pipeline and get/post/delete with it.

I cant just arbitrarily assume you'll be sending JSON and therefore json.dumps(payload).

If you want to urlencode, the app ships with a urlencode command as well.

It can be tricky getting the command to work but the pay off is good too 🙂

Try adding debug=t to the command as well... it can help to see what it receives as inputs.

0 Karma

ian_thomas
Path Finder

We need to do change the uri as well which is why we went to map. Any tips on how to sub a field into the uri argument welcome.

0 Karma

jkat54
SplunkTrust
SplunkTrust

We did it in this example after someone came up with the use case:

Deleting fired alerts for search named "Test Alert":
| rest /servicesNS/admin/search/alerts/fired_alerts/Test%20Alert | fields title | head 10 | map search="|curl method=delete uri="https://localhost:8089/servicesNS/admin/search/alerts/fired_alerts/$title$" user=admin pass=changeme | table *"

As you can see, we take the random search ids (title) from the splunk rest, and then map it to the URI used in the curl command in order to delete triggered alerts via the curl command.

0 Karma

jkat54
SplunkTrust
SplunkTrust

i see you're getting really close in one of your screenshots

0 Karma

jkat54
SplunkTrust
SplunkTrust

instead of making one field that maps through try more than one...

one for URI and another for data

| map search="|curl uri=$URI$ data=$PAYLOAD$" for example

phastings
Explorer

Thanks for the quick reply, and support on this matter.

I tried using two variables to map this instead of one concatenated variable and I wasn't able to find success with that either.
Its also worth noting I tried using single quotes in the JSON to no avail.

Current_status

0 Karma

jkat54
SplunkTrust
SplunkTrust

Can you paste the search somewhere that I can copy it out an try to replicate the same on my end?

0 Karma

phastings
Explorer

Absolutely. Below is the current search. It is in the context of ITSI but the principal JSON as payload should be the same.

Thank you.

| rest splunk_server=local /servicesNS/nobody/SA-ITOA/itoa_interface/service fields="_key,entity_rules,title"
| eval strippedValue=spath(value,"{}")
| mvexpand strippedValue
`comment("breaking up JSON body into variables we can use to reusable fields")`
| eval service_id=spath(strippedValue,"_key"), service_title=spath(strippedValue,"title"), entity_rules_field=spath(strippedValue,"entity_rules{}"), newestValue=spath(strippedValue,"kpis{}")
`comment("return all results where the entity rules have not been applied")`
| search NOT entity_rules_field="*"
| eval comment3="adding service ID to act as a lookup key, to the URL"
| eval curl_command="https://localhost:8089/servicesNS/nobody/SA-ITOA/itoa_interface/service/".service_id
| eval comment4="adding partial data flag to the existing url. We want to only update the desired entity_rules today"
| eval curl_command=curl_command."/?is_partial_data=1"
`comment("adding data body flag with JSON body, there is a space to separate the URL from the data flag")`
| eval curl_command=curl_command
| eval dataInput="{\"entity_rules\": [{\"rule_items\": [{\"field_type\": \"info\", \"field\": \"parentserviceinfo\", \"rule_type\": \"matches\", \"value\": \"deletemeparentservice-dc100\"}], \"rule_condition\": \"AND\"}], \"permissions\": {\"read\": true, \"group\": {\"read\": true, \"delete\": true, \"write\": true}, \"user\": \"admin\", \"delete\": true, \"write\": true}, \"object_type\": \"service\", \"sec_grp\": \"default_itsi_security_group\"}"
| map search="| curl method=post uri=$curl_command$ data=$dataInput$ user=admin pass=omitted debug=t"
0 Karma

ian_thomas
Path Finder

Good idea.

0 Karma

ian_thomas
Path Finder

Can you post version of CLI curl that successfully updated ITSI entity rule? That way we will know what worked. Thanks.

0 Karma

phastings
Explorer

This is the working command via bash
https://pastebin.com/WhkMNj2G

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...