Getting Data In

How to upsert to the KV Store via REST API?

sjodle
Path Finder

http://dev.splunk.com/view/SP-CAAAEY7 states that the KV store allows "upserts", but doesn't elaborate on how to perform them. Does this mean that I can perform an operation that inserts a document into the collection, or updates it if it already exists? How can I do this?

In my tests, POSTing a document with a duplicate _key to storage/collections/data/{collection} causes an error, and POSTing an update to storage/collections/data/{collection}/{id} where id doesn't exist also returns an error. Is there a single operation that's equivalent to SQL's INSERT ... ON DUPLICATE KEY UPDATE?

0 Karma
1 Solution

dgladkikh_splun
Splunk Employee
Splunk Employee

@sjodle, this can be done using batch_save endpoint http://docs.splunk.com/Documentation/Splunk/6.3.2/RESTREF/RESTkvstore#storage.2Fcollections.2Fdata.2...

You can find few examples on this page as well http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZG

As suggested by @Murali2888 - batch_save is very similar to SPL command outputlookup append=true, it inserts all new documents and replaces all documents which already exist in collection.

View solution in original post

dgladkikh_splun
Splunk Employee
Splunk Employee

@sjodle, this can be done using batch_save endpoint http://docs.splunk.com/Documentation/Splunk/6.3.2/RESTREF/RESTkvstore#storage.2Fcollections.2Fdata.2...

You can find few examples on this page as well http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZG

As suggested by @Murali2888 - batch_save is very similar to SPL command outputlookup append=true, it inserts all new documents and replaces all documents which already exist in collection.

sjodle
Path Finder

Thanks! I'll check that out!

0 Karma

Murali2888
Communicator

Hi sjodle,

This may not be what you are looking for, but an alternative option would be to use lookup/outputlookup command.
Below query would do the upserts into kvstore

Base Search | lookup <lookup_name> <lookup_field> OUTPUTNEW _key AS _key | outputlookup <lookup_name> append=true
0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Hi @sjodle,

You can create a new item by POSTing to the collection. This POST allocates the /{id} resource that you can then use for updating that item, using a POST to the /{id} endpoint.

In case it helps, having the "upsert" functionality might allow for mistakes where a user means to indicate an existing id but makes a POST using an incorrect id that happens to be non-allocated. If a new resource is created for the incorrect and non-allocated id, then the update can't be made to the correct id and the user might not find out.

Hope this helps!

sjodle
Path Finder

In case it helps, having the "upsert" functionality might allow for mistakes where a user means to indicate an existing id but makes a POST using an incorrect id that happens to be non-allocated. If a new resource is created for the incorrect and non-allocated id, then the update can't be made to the correct id and the user might not find out.

MongoDB (which the KV Store is built upon) mitigates this problem by requiring an "upsert: true" parameter to be explicitly passed, otherwise an error is returned. This ensures that the user understands the potential risks associated with upserts, and ensures that they don't happen by default without the user's knowledge.

sjodle
Path Finder

Thanks for the answer. I'll mark it as answered and proceed with my plan B: try an insert, except if it fails due to duplicate key, retry it as an update. Upserts would be an excellent feature addition in future versions, though!

frobinson_splun
Splunk Employee
Splunk Employee

Thanks for your suggestions, @sjodle! I've passed them along to the engineering team.

Get Updates on the Splunk Community!

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...