Attempting to restore a KVStore collection: has anyone seen or successfully troubleshooted the following error?

[pabdola]

Hi Everyone,

I am currently trying to achieve a quite simple process: set up a scalable way to backup/restore some KVStore collections from production Splunk servers.

Following the appropriate Splunk documentation (https://docs.splunk.com/Documentation/Splunk/7.1.2/Admin/BackupKVstore), i was able to successfully backup my collections in JSON formatted files.

However, when i try to restore them to the same production server, it fails with the following errors (from /opt/splunk/var/log/splunkd.log):

splunk: ERROR 1535113443.183 KVStorageProvider - An error occurred during the last operation ('dropCollection', domain: '5', code: '26'): ns not found
splunk: WARN  1535113443.188 KVStoreAdminHandler - No data found to restore matching specified parameters archiveName="backupfile.tar.gz", appName="all apps", collectionName="collection"
splunk: ERROR 1535113443.188 KVStoreAdminHandler - \n

This seems quite cryptic to me. I am wondering whether anyone could have encounter a similar issue or error message, or could know some troubleshooting tips that will help me solving this?

[pabdola · Accepted solution]

Just in case someone faces the same issue that i did, here is a fix that Andrew from Splunk Support gave me.
Just edit the following stanza/parameter into your "limits.conf" configuration file:

[kvstore]
max_documents_per_batch_save = <unsigned int>
* The maximum number of documents that can be saved in a single batch
* Default: 1000

After raising this "max_documents_per_batch_save" parameter to a value superior to the number of entries found into your Collection, you should be able to restore it properly.

[pabdola]

In addition to the modification above, if the number of entry in you collection exceeds 50000, another "limits.conf" parameter must be modified:

 [kvstore]
max_rows_per_query = <unsigned int>
* The maximum number of rows that will be returned for a single query to 
  a collection.
* If the query returns more rows than the specified value, then returned 
  result set will contain the number of rows specified in this value.
* Default: 50000

Otherwise, the restored KVstore collection will be truncated to 50000 entries.

[DavidHourani]

Try this !

https://splunkbase.splunk.com/app/3536/

It's way easier to use!

Cheers,
D

[pabdola]

Thanks for your reply David. This application seems interesting, but it seems very WebUI-focused, whereas i am trying to use CLI as much as possible, in order to be able to automate things. It is always good to discover new applications though, so i will probably give a go to this app! 🙂

[DavidHourani]

yeah for CLI go for commands added after 7.0.

The app gives you advanced search commands queries only 😄 pretty useful for backing up and importing kv back into SHC.

[pabdola]

Yes, i was trying to use those commands, but was hitting a number of entries limit (1000) while using the "restore kvstore" CLI command. A fix was found thanks to Splunk Support as described in my answer below.

It is surely interesting to be able to trigger backup/restore directly from WebUI using SPL.