Knowledge Management

Can you help me with the following KV Store error: "Detected unclean shutdown - /home/dbindex/kvstore/mongo/mongod.lock is not empty"

paola92
Explorer

Hi,

I have several errors related to KV Store as:

-Failed to start KV Store process. See mongod.log and splunkd.log for details.
-KV Store changed status to failed. KVStore process terminated.
-KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details.

So when reviewing the mongod.log I see:

Detected unclean shutdown - /home/dbindex/kvstore/mongo/mongod.lock is not empty.
 I STORAGE  [initandlisten]
 I STORAGE  [initandlisten] ** WARNING: Readahead for /home/dbindex/kvstore/mongo is set to 4096KB
 I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
 I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
 I STORAGE  [initandlisten] **************
 old lock file: /home/dbindex/kvstore/mongo/mongod.lock.  probably means unclean shutdown,
 but there are no journal files to recover.
 this is likely human error or filesystem corruption.
 please make sure that your journal directory is mounted.
 found 76 dbs.
 see: http://dochub.mongodb.org/core/repair for more information

And, I tried to run the ./mongod --dbpath /DB/kvstore/mongo --repair and obtained the error:

./mongod: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

So I need help to solve the problem!

0 Karma
1 Solution

reswob4
Builder

The answer by @claudio.manig in this thread https://answers.splunk.com/answers/236495/splunk-kv-store-does-not-start.html helped me.

After verifying all the permissions discussed were set correctly on my server (linux), I stopped Splunk, deleted the mongod.lock file and restarted splunk and the error went away. I didn't need to run the repair utility.

View solution in original post

rajan_kumar_rai
Loves-to-Learn

We have to delete the mongod.lock file from the /Application/Splunk/var/lib/kvstore/mongo —> mongod.lock

We have to remove this file to done that simply run below command in that path

# SUDO rm -rf mongod.lock >> it will delete the that file.

Next we have to provide the read permission to SPLUNK.KEY file to done that run below command

# CHMOD 400 SPLUNK.KEY ( IT WORKS ) 

Then restart the splunk from GUI. Go to settings > server controls > Restart 😀

0 Karma

reswob4
Builder

The answer by @claudio.manig in this thread https://answers.splunk.com/answers/236495/splunk-kv-store-does-not-start.html helped me.

After verifying all the permissions discussed were set correctly on my server (linux), I stopped Splunk, deleted the mongod.lock file and restarted splunk and the error went away. I didn't need to run the repair utility.

Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...