All Apps and Add-ons

KVStore mongo db failing to start, after kvstore is restored from backup

splunkIT
Splunk Employee
Splunk Employee

I have restored my kvstore from backup using the steps from here: https://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/BackupKVstore

But now the SHC node's kvstore cannot start. Here are the messages seen in mongod.log:
2018-01-19T20:34:36.865Z I JOURNAL [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal
2018-01-19T20:34:36.865Z I JOURNAL [initandlisten] recover begin
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] info no lsn file in journal/ directory
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] recover lsn: 0
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] recover /opt/splunk/var/lib/splunk/kvstore/mongo/journal/j._0
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] recover applying initial journal section with sequence number 65
2018-01-19T20:34:36.866Z F JOURNAL [initandlisten] dbexception during recovery: 15923 couldn't get file length when opening mapping /opt/splunkvar/lib/splunk/kvstore/mongo/local.0 boost::filesystem::file_size: No such file or directory: "/opt/splunkvar/lib/splunk/kvstore/mongo/local.0"
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] exception in initAndListen: 15923 couldn't get file length when opening mapping /opt/splunk/var/lib/splunk/kvstore/mongo/local.0 boost::filesystem::file_size: No such file or directory: "/opt/splunk/var/lib/splunk/kvstore/mongo/local.0", terminating
2018-01-19T20:34:36.866Z W REPL [initandlisten] ReplicationCoordinatorImpl::shutdown() called before startReplication() finished. Shutting down without cleaning up the replication system
2018-01-19T20:34:36.866Z I CONTROL [initandlisten] now exiting
2018-01-19T20:34:36.866Z I NETWORK [initandlisten] shutdown: going to close listening sockets...
2018-01-19T20:34:36.866Z I NETWORK [initandlisten] shutdown: going to flush diaglog...
2018-01-19T20:34:36.866Z I NETWORK [initandlisten] shutdown: going to close sockets...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] shutdown: waiting for fs preallocator...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] shutdown: final commit...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] shutdown: closing all files...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] closeAllFiles() finished

Tags (1)
0 Karma

scheng_splunk
Splunk Employee
Splunk Employee

looks like it's missing /splunk/var/lib/splunk/kvstore/mongo/local.* file which stores the cluster information.
You may want to check if you have /opt/splunkvar/lib/splunk/kvstore/mongo/local.0 file existing from your kvstore backup.

Also if by any chance you did run "splunk clean kvstore --cluster" command after restoring the kvstore backup, it would wipe out all the local.* files which contains the cluster information.

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...