topic How to upsert to the KV Store via REST API? in Getting Data In

How to upsert to the KV Store via REST API?

sjodle — Tue, 19 Jan 2016 22:09:58 GMT

http://dev.splunk.com/view/SP-CAAAEY7 states that the KV store allows "upserts", but doesn't elaborate on how to perform them. Does this mean that I can perform an operation that inserts a document into the collection, or updates it if it already exists? How can I do this?

In my tests, POSTing a document with a duplicate _key to storage/collections/data/{collection} causes an error, and POSTing an update to storage/collections/data/{collection}/{id} where id doesn't exist also returns an error. Is there a single operation that's equivalent to SQL's INSERT ... ON DUPLICATE KEY UPDATE?

Re: How to upsert to the KV Store via REST API?

frobinson_splun — Tue, 19 Jan 2016 22:57:38 GMT

Hi @sjodle,

You can create a new item by POSTing to the collection. This POST allocates the /{id} resource that you can then use for updating that item, using a POST to the /{id} endpoint.

In case it helps, having the "upsert" functionality might allow for mistakes where a user means to indicate an existing id but makes a POST using an incorrect id that happens to be non-allocated. If a new resource is created for the incorrect and non-allocated id, then the update can't be made to the correct id and the user might not find out.

Hope this helps!

Re: How to upsert to the KV Store via REST API?

Murali2888 — Tue, 19 Jan 2016 23:48:22 GMT

Hi sjodle,

This may not be what you are looking for, but an alternative option would be to use lookup/outputlookup command.
Below query would do the upserts into kvstore

Base Search | lookup <lookup_name> <lookup_field> OUTPUTNEW _key AS _key | outputlookup <lookup_name> append=true

Re: How to upsert to the KV Store via REST API?

sjodle — Wed, 20 Jan 2016 18:29:52 GMT

In case it helps, having the "upsert" functionality might allow for mistakes where a user means to indicate an existing id but makes a POST using an incorrect id that happens to be non-allocated. If a new resource is created for the incorrect and non-allocated id, then the update can't be made to the correct id and the user might not find out.

MongoDB (which the KV Store is built upon) mitigates this problem by requiring an "upsert: true" parameter to be explicitly passed, otherwise an error is returned. This ensures that the user understands the potential risks associated with upserts, and ensures that they don't happen by default without the user's knowledge.

Re: How to upsert to the KV Store via REST API?

sjodle — Wed, 20 Jan 2016 18:33:45 GMT

Thanks for the answer. I'll mark it as answered and proceed with my plan B: try an insert, except if it fails due to duplicate key, retry it as an update. Upserts would be an excellent feature addition in future versions, though!

Re: How to upsert to the KV Store via REST API?

frobinson_splun — Wed, 20 Jan 2016 18:54:47 GMT

Thanks for your suggestions, @sjodle! I've passed them along to the engineering team.

Re: How to upsert to the KV Store via REST API?

dgladkikh_splun — Tue, 29 Sep 2020 08:29:27 GMT

@sjodle, this can be done using batch_save endpoint http://docs.splunk.com/Documentation/Splunk/6.3.2/RESTREF/RESTkvstore#storage.2Fcollections.2Fdata.2F.7Bcollection.7D.2Fbatch_save

You can find few examples on this page as well http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZG

As suggested by @Murali2888 - batch_save is very similar to SPL command outputlookup append=true, it inserts all new documents and replaces all documents which already exist in collection.

Re: How to upsert to the KV Store via REST API?

sjodle — Wed, 20 Jan 2016 21:29:49 GMT

Thanks! I'll check that out!