How to Restore KVStore if we do not have splunk kvstore backup

[vksplunk1]

Hi - We have accidentally deleted kvstore with outputlookup command. We do not have a backup from splunk.

How to Restore KVStore from back up of  splunk home( /opt/splunk )directory backup

[kiran_panchavat]

@vksplunk1 

The KV store isn’t very reliable, so it's best to back it up regularly.

1. Some apps store their lookups in the kvstore. (collections.conf)

2. Some apps store all their configuration in the kvstore (ITSI, but they also do daily backups.

For Splunk itself

1. It sometimes uses the kvstore to track which summary indexing time range was done.

It's wise to back up your KV store regularly since it's vulnerable to data loss. If it gets corrupted, deleted, or runs into issues during an upgrade or restart, you could lose valuable data. Keeping backups helps you recover your data quickly if anything goes wrong.

https://community.splunk.com/t5/Knowledge-Management/Is-there-any-way-to-retrieve-kv-store-that-was-... 

[livehybrid]

Hi @vksplunk1 

By default your KV store files will be stored in $SPLUNK_HOME/var/lib/splunk/kvstore/mongo - so if you have a backup of this directory you may be able to get the data back based on the time it was backed up, however I would look at recovering this to a different / test server rather than your production instance as it isnt possible to pick and choose which files to restore. 

Therefore you might need to recover the whole backup and then take a backup from the recovered data before restoring. Do you have other lookups also? This will affect those if you overwrite from an old backup.

You could try this approach, and depending on the size of your lost KV Store lookup. you could export it from the restored backup, then load it back into the KV Store on your production instance using a mixture of |inputlookup <restoredData.csv> | outputlookup <OriginalLookupName>

Do you think this might work for your situation?

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[vksplunk1]

Thank you for your suggestions. We do not have a test server to restore before restoring it on to prod server. 

On a separate note, is it possible to schedule a report or a script to backup kvstore on a daily basis to avoid restoring from  backup of /opt/splunk/var/lib/splunk/backup  directory