Getting Data In

How to upsert to the KV Store via REST API?

Path Finder

http://dev.splunk.com/view/SP-CAAAEY7 states that the KV store allows "upserts", but doesn't elaborate on how to perform them. Does this mean that I can perform an operation that inserts a document into the collection, or updates it if it already exists? How can I do this?

In my tests, POSTing a document with a duplicate _key to storage/collections/data/{collection} causes an error, and POSTing an update to storage/collections/data/{collection}/{id} where id doesn't exist also returns an error. Is there a single operation that's equivalent to SQL's INSERT ... ON DUPLICATE KEY UPDATE?

0 Karma
1 Solution

Splunk Employee
Splunk Employee

@sjodle, this can be done using batch_save endpoint http://docs.splunk.com/Documentation/Splunk/6.3.2/RESTREF/RESTkvstore#storage.2Fcollections.2Fdata.2...

You can find few examples on this page as well http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZG

As suggested by @Murali2888 - batch_save is very similar to SPL command outputlookup append=true, it inserts all new documents and replaces all documents which already exist in collection.

View solution in original post

Splunk Employee
Splunk Employee

@sjodle, this can be done using batch_save endpoint http://docs.splunk.com/Documentation/Splunk/6.3.2/RESTREF/RESTkvstore#storage.2Fcollections.2Fdata.2...

You can find few examples on this page as well http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZG

As suggested by @Murali2888 - batch_save is very similar to SPL command outputlookup append=true, it inserts all new documents and replaces all documents which already exist in collection.

View solution in original post

Path Finder

Thanks! I'll check that out!

0 Karma

Communicator

Hi sjodle,

This may not be what you are looking for, but an alternative option would be to use lookup/outputlookup command.
Below query would do the upserts into kvstore

Base Search | lookup <lookup_name> <lookup_field> OUTPUTNEW _key AS _key | outputlookup <lookup_name> append=true
0 Karma

Splunk Employee
Splunk Employee

Hi @sjodle,

You can create a new item by POSTing to the collection. This POST allocates the /{id} resource that you can then use for updating that item, using a POST to the /{id} endpoint.

In case it helps, having the "upsert" functionality might allow for mistakes where a user means to indicate an existing id but makes a POST using an incorrect id that happens to be non-allocated. If a new resource is created for the incorrect and non-allocated id, then the update can't be made to the correct id and the user might not find out.

Hope this helps!

Path Finder

In case it helps, having the "upsert" functionality might allow for mistakes where a user means to indicate an existing id but makes a POST using an incorrect id that happens to be non-allocated. If a new resource is created for the incorrect and non-allocated id, then the update can't be made to the correct id and the user might not find out.

MongoDB (which the KV Store is built upon) mitigates this problem by requiring an "upsert: true" parameter to be explicitly passed, otherwise an error is returned. This ensures that the user understands the potential risks associated with upserts, and ensures that they don't happen by default without the user's knowledge.

Path Finder

Thanks for the answer. I'll mark it as answered and proceed with my plan B: try an insert, except if it fails due to duplicate key, retry it as an update. Upserts would be an excellent feature addition in future versions, though!

Splunk Employee
Splunk Employee

Thanks for your suggestions, @sjodle! I've passed them along to the engineering team.

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!