KVStore mongo db failing to start, after kvstore is restored from backup

splunkIT — Tue, 29 Sep 2020 17:41:44 GMT

I have restored my kvstore from backup using the steps from here: https://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/BackupKVstore

But now the SHC node's kvstore cannot start. Here are the messages seen in mongod.log:
2018-01-19T20:34:36.865Z I JOURNAL [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal
2018-01-19T20:34:36.865Z I JOURNAL [initandlisten] recover begin
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] info no lsn file in journal/ directory
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] recover lsn: 0
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] recover /opt/splunk/var/lib/splunk/kvstore/mongo/journal/j._0
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] recover applying initial journal section with sequence number 65
2018-01-19T20:34:36.866Z F JOURNAL [initandlisten] dbexception during recovery: 15923 couldn't get file length when opening mapping /opt/splunkvar/lib/splunk/kvstore/mongo/local.0 boost::filesystem::file_size: No such file or directory: "/opt/splunkvar/lib/splunk/kvstore/mongo/local.0"
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] exception in initAndListen: 15923 couldn't get file length when opening mapping /opt/splunk/var/lib/splunk/kvstore/mongo/local.0 boost::filesystem::file_size: No such file or directory: "/opt/splunk/var/lib/splunk/kvstore/mongo/local.0", terminating
2018-01-19T20:34:36.866Z W REPL [initandlisten] ReplicationCoordinatorImpl::shutdown() called before startReplication() finished. Shutting down without cleaning up the replication system
2018-01-19T20:34:36.866Z I CONTROL [initandlisten] now exiting
2018-01-19T20:34:36.866Z I NETWORK [initandlisten] shutdown: going to close listening sockets...
2018-01-19T20:34:36.866Z I NETWORK [initandlisten] shutdown: going to flush diaglog...
2018-01-19T20:34:36.866Z I NETWORK [initandlisten] shutdown: going to close sockets...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] shutdown: waiting for fs preallocator...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] shutdown: final commit...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] shutdown: closing all files...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] closeAllFiles() finished

Re: KVStore mongo db failing to start, after kvstore is restored from backup

scheng_splunk — Thu, 16 Aug 2018 08:09:38 GMT

looks like it's missing /splunk/var/lib/splunk/kvstore/mongo/local.* file which stores the cluster information.
You may want to check if you have /opt/splunkvar/lib/splunk/kvstore/mongo/local.0 file existing from your kvstore backup.

Also if by any chance you did run "splunk clean kvstore --cluster" command after restoring the kvstore backup, it would wipe out all the local.* files which contains the cluster information.

topic KVStore mongo db failing to start, after kvstore is restored from backup in All Apps and Add-ons

KVStore mongo db failing to start, after kvstore is restored from backup

Re: KVStore mongo db failing to start, after kvstore is restored from backup