Knowledge Management

Can you help me with the following KV Store error: "Detected unclean shutdown - /home/dbindex/kvstore/mongo/mongod.lock is not empty"

paola92
Explorer

Hi,

I have several errors related to KV Store as:

-Failed to start KV Store process. See mongod.log and splunkd.log for details.
-KV Store changed status to failed. KVStore process terminated.
-KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details.

So when reviewing the mongod.log I see:

Detected unclean shutdown - /home/dbindex/kvstore/mongo/mongod.lock is not empty.
 I STORAGE  [initandlisten]
 I STORAGE  [initandlisten] ** WARNING: Readahead for /home/dbindex/kvstore/mongo is set to 4096KB
 I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
 I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
 I STORAGE  [initandlisten] **************
 old lock file: /home/dbindex/kvstore/mongo/mongod.lock.  probably means unclean shutdown,
 but there are no journal files to recover.
 this is likely human error or filesystem corruption.
 please make sure that your journal directory is mounted.
 found 76 dbs.
 see: http://dochub.mongodb.org/core/repair for more information

And, I tried to run the ./mongod --dbpath /DB/kvstore/mongo --repair and obtained the error:

./mongod: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

So I need help to solve the problem!

0 Karma
1 Solution

reswob4
Builder

The answer by @claudio.manig in this thread https://answers.splunk.com/answers/236495/splunk-kv-store-does-not-start.html helped me.

After verifying all the permissions discussed were set correctly on my server (linux), I stopped Splunk, deleted the mongod.lock file and restarted splunk and the error went away. I didn't need to run the repair utility.

View solution in original post

rajan_kumar_rai
Loves-to-Learn

We have to delete the mongod.lock file from the /Application/Splunk/var/lib/kvstore/mongo —> mongod.lock

We have to remove this file to done that simply run below command in that path

# SUDO rm -rf mongod.lock >> it will delete the that file.

Next we have to provide the read permission to SPLUNK.KEY file to done that run below command

# CHMOD 400 SPLUNK.KEY ( IT WORKS ) 

Then restart the splunk from GUI. Go to settings > server controls > Restart 😀

0 Karma

reswob4
Builder

The answer by @claudio.manig in this thread https://answers.splunk.com/answers/236495/splunk-kv-store-does-not-start.html helped me.

After verifying all the permissions discussed were set correctly on my server (linux), I stopped Splunk, deleted the mongod.lock file and restarted splunk and the error went away. I didn't need to run the repair utility.

Get Updates on the Splunk Community!

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...

Want to Reduce Costs, Mitigate Risk, Improve Performance, or Increase Efficiencies? ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...