topic Re: KV store keys, REST API and URL encoding forward slashes in Getting Data In

KV store keys, REST API and URL encoding forward slashes

hughkelley — Thu, 26 Dec 2019 21:11:20 GMT

I have a KV collection that uses a CIDR-style network address as the key value. This means that delete operations have to send this
forward slash in the URL. I'm getting a 404 "Could not find object".

For the key "1.1.1.1/29" I'm using this URL:

/servicesNS/nobody/search/storage/collections/data/collectionName/1.1.1.1%2f29

Update operations work fine (because the slash is in the POST data, not the URL?).

What (other?) encoding/escaping do I need? On a whim, I tried doubling the forward slashes, but no dice.

Re: KV store keys, REST API and URL encoding forward slashes

mydog8it — Fri, 27 Dec 2019 17:50:14 GMT

The documentation -->> https://docs.splunk.com/Documentation/SplunkCloud/8.0.0/RESTREF/RESTkvstore
eludes to the CIDR being translated to Canonical style record when it is recorded in the KV Store.

If that is true, the 1.1.1.1/29 record you are trying to delete would be recorded as 001.001.001.001/29 in the KV Store.

Try running something like this to see how the data is formatted:

curl -k -u admin:yourpassword \   https://localhost:8089/servicesNS/nobody/kvstoretest/storage/collections/data/kvstorecoll

The other thought I had was passing the key as data, as seen for save operations on the dev site-->> https://dev.splunk.com/enterprise/docs/developapps/kvstore/usetherestapitomanagekv/

I was thinking delete might work this way, but I don't see it documented anywhere.

 curl -k -u admin:yourpassword -X DELETE \  
 https://localhost:8089//servicesNS/nobody/search/storage/collections/data/collectionName \
        -H 'Content-Type: application/json' \
        -d '[{"name": "001.001.001.001/29"}]'

Hope this helps!

Re: KV store keys, REST API and URL encoding forward slashes

hughkelley — Fri, 27 Dec 2019 18:43:34 GMT

Unfortunately, no change in behavior. Still a 404 with this format.

/servicesNS/nobody/search/storage/collections/data/networks/010.009.009.009%2f29

Re: KV store keys, REST API and URL encoding forward slashes

ryanoconnor — Wed, 30 Sep 2020 03:33:21 GMT

For this, you're going to want to follow a slightly different strategy as CIDR may not be unique in your KVStore, but the Key ID always will be.

So instead of directly attempting to post the CIDR in the URL, you can add one extra step.

Query for any records where your CIDR matches. An example I've used is curl -k -u admin:password https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll?query=%7B%22cidr%22%3A%20%221.2.4.7%2F24%22%7D
This will give you data back that shows all records with this CIDR. From there you can get the relevant key id's that you want to delete:

[ { "name" : "Ryan", "cidr" : "1.2.4.7/24", "cidrnoslash" : "1.1.1.1", "_user" : "nobody", "_key" : "5e174b07e2edd2a9e9218e5a" } ]

You can then post a similar delete command, except with the Key ID, such as: curl -k -u admin:password -X DELETE https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll/5e174b07e2edd2a9e9218e5a -H 'Content-Type: application/json'

Hope this helps!

Re: KV store keys, REST API and URL encoding forward slashes

hughkelley — Thu, 09 Jan 2020 16:48:29 GMT

Unfortunately, I use the CIDR address as the _key value. This was necessary so that I didn't have to track a separate ID. When I use the query approach you describe I get the key in plain CIDR format.

netCidr    : 99.99.6.0/23
_user      : nobody
_key       : 99.99.6.0/23

I'm thinking this means I need to find a supported way to encode it.

Re: KV store keys, REST API and URL encoding forward slashes

ryanoconnor — Wed, 30 Sep 2020 03:34:20 GMT

Granted, it's probably best to leave _key as something unique. That's usually a reserved field with a unique ID for each record (https://dev.splunk.com/enterprise/docs/developapps/kvstore/aboutkvstorecollections/).

You still should be able to make this work I believe. This is on my test instance, Splunk 8.0.1. Here is my sample commands and output:

curl -k -u admin:password  https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll

curl -k -u admin:password https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll -H 'Content-Type: application/json' -d  '{"name": "Ryan", "netCidr": "'"99.99.6.0/23"'", "_key": "99.99.6.0/23"}'

curl -k -u admin:password https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll -H 'Content-Type: application/json' -d  '{"name": "Ryan", "netCidr": "'"88.88.6.0/24"'", "_key": "88.88.6.0/24"}'

curl -k -u admin:password  https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll

curl -k -u admin:password -X DELETE https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll/99.99.6.0%2f23 -H 'Content-Type: application/json'

curl -k -u admin:password https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll

Output:

ryan$ curl -k -u admin:password  https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll
[ ]
ryan$ curl -k -u admin:password https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll -H 'Content-Type: application/json' -d '{"name": "Ryan", "netCidr": "'"99.99./23"'", "_key": "99.99.6.0/23"}'
{"_key":"99.99.6.0/23"}

ryan$ curl -k -u admin:password https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll -H 'Content-Type: application/json' -d '{"name": "Ryan", /24"'", "_key": "88.88.6.0/24"}'
{"_key":"88.88.6.0/24"}

ryan$ curl -k -u admin:password https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll
[ { "name" : "Ryan", "netCidr" : "88.88.6.0/24", "_user" : "nobody", "_key" : "88.88.6.0/24" }, { "name" : "Ryan", "netCidr" : "99.99.6.0/23", "_user" : "nobody", "_key" : "99.99.6.0/23" } ]

ryan$ curl -k -u admin:password -X DELETE https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll/99.99.6.0%2f23 -H 'Content-Type: application/json'

ryan$ curl -k -u admin:password https://localhost:8089/servicesNS/nobody/search/storage/collections/data/kvstorecoll
[ { "name" : "Ryan", "netCidr" : "88.88.6.0/24", "_user" : "nobody", "_key" : "88.88.6.0/24" }]

Re: KV store keys, REST API and URL encoding forward slashes

hughkelley — Fri, 10 Jan 2020 17:27:03 GMT

That bring us full circle. This URL:

https://xxx/servicesNS/nobody/search/storage/collections/data/networks/99.99.6.0%2f23

gets me this result:

WebCmdletWebResponseException

    Could not find object.

The remote server returned an error: (404) Not Found.

I know the _key is in the collection because I do a full fetch of the collection at the beginning of the script. This works with other types of data (usernames and other _key data) - just not these addresses.