Knowledge Management

Attempting to restore a KVStore collection: has anyone seen or successfully troubleshooted the following error?

pabdola
Explorer

Hi Everyone,

I am currently trying to achieve a quite simple process: set up a scalable way to backup/restore some KVStore collections from production Splunk servers.

Following the appropriate Splunk documentation (https://docs.splunk.com/Documentation/Splunk/7.1.2/Admin/BackupKVstore), i was able to successfully backup my collections in JSON formatted files.

However, when i try to restore them to the same production server, it fails with the following errors (from /opt/splunk/var/log/splunkd.log):

splunk: ERROR 1535113443.183 KVStorageProvider - An error occurred during the last operation ('dropCollection', domain: '5', code: '26'): ns not found
splunk: WARN  1535113443.188 KVStoreAdminHandler - No data found to restore matching specified parameters archiveName="backupfile.tar.gz", appName="all apps", collectionName="collection"
splunk: ERROR 1535113443.188 KVStoreAdminHandler - \n

This seems quite cryptic to me. I am wondering whether anyone could have encounter a similar issue or error message, or could know some troubleshooting tips that will help me solving this?

0 Karma
1 Solution

pabdola
Explorer

Just in case someone faces the same issue that i did, here is a fix that Andrew from Splunk Support gave me.
Just edit the following stanza/parameter into your "limits.conf" configuration file:

[kvstore]
max_documents_per_batch_save = <unsigned int>
* The maximum number of documents that can be saved in a single batch
* Default: 1000

After raising this "max_documents_per_batch_save" parameter to a value superior to the number of entries found into your Collection, you should be able to restore it properly.

View solution in original post

0 Karma

pabdola
Explorer

Just in case someone faces the same issue that i did, here is a fix that Andrew from Splunk Support gave me.
Just edit the following stanza/parameter into your "limits.conf" configuration file:

[kvstore]
max_documents_per_batch_save = <unsigned int>
* The maximum number of documents that can be saved in a single batch
* Default: 1000

After raising this "max_documents_per_batch_save" parameter to a value superior to the number of entries found into your Collection, you should be able to restore it properly.

0 Karma

pabdola
Explorer

In addition to the modification above, if the number of entry in you collection exceeds 50000, another "limits.conf" parameter must be modified:

 [kvstore]
max_rows_per_query = <unsigned int>
* The maximum number of rows that will be returned for a single query to 
  a collection.
* If the query returns more rows than the specified value, then returned 
  result set will contain the number of rows specified in this value.
* Default: 50000

Otherwise, the restored KVstore collection will be truncated to 50000 entries.

0 Karma

DavidHourani
Super Champion

Try this !

https://splunkbase.splunk.com/app/3536/

It's way easier to use!

Cheers,
D

pabdola
Explorer

Thanks for your reply David. This application seems interesting, but it seems very WebUI-focused, whereas i am trying to use CLI as much as possible, in order to be able to automate things. It is always good to discover new applications though, so i will probably give a go to this app! 🙂

DavidHourani
Super Champion

yeah for CLI go for commands added after 7.0.

The app gives you advanced search commands queries only 😄 pretty useful for backing up and importing kv back into SHC.

0 Karma

pabdola
Explorer

Yes, i was trying to use those commands, but was hitting a number of entries limit (1000) while using the "restore kvstore" CLI command. A fix was found thanks to Splunk Support as described in my answer below.

It is surely interesting to be able to trigger backup/restore directly from WebUI using SPL.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...