Knowledge Management

Can you help me with the following KV Store error: "Detected unclean shutdown - /home/dbindex/kvstore/mongo/mongod.lock is not empty"

paola92
Explorer

Hi,

I have several errors related to KV Store as:

-Failed to start KV Store process. See mongod.log and splunkd.log for details.
-KV Store changed status to failed. KVStore process terminated.
-KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details.

So when reviewing the mongod.log I see:

Detected unclean shutdown - /home/dbindex/kvstore/mongo/mongod.lock is not empty.
 I STORAGE  [initandlisten]
 I STORAGE  [initandlisten] ** WARNING: Readahead for /home/dbindex/kvstore/mongo is set to 4096KB
 I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
 I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
 I STORAGE  [initandlisten] **************
 old lock file: /home/dbindex/kvstore/mongo/mongod.lock.  probably means unclean shutdown,
 but there are no journal files to recover.
 this is likely human error or filesystem corruption.
 please make sure that your journal directory is mounted.
 found 76 dbs.
 see: http://dochub.mongodb.org/core/repair for more information

And, I tried to run the ./mongod --dbpath /DB/kvstore/mongo --repair and obtained the error:

./mongod: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

So I need help to solve the problem!

0 Karma
1 Solution

reswob4
Builder

The answer by @claudio.manig in this thread https://answers.splunk.com/answers/236495/splunk-kv-store-does-not-start.html helped me.

After verifying all the permissions discussed were set correctly on my server (linux), I stopped Splunk, deleted the mongod.lock file and restarted splunk and the error went away. I didn't need to run the repair utility.

View solution in original post

rajan_kumar_rai
Loves-to-Learn Lots

We have to delete the mongod.lock file from the /Application/Splunk/var/lib/kvstore/mongo —> mongod.lock

We have to remove this file to done that simply run below command in that path

# SUDO rm -rf mongod.lock >> it will delete the that file.

Next we have to provide the read permission to SPLUNK.KEY file to done that run below command

# CHMOD 400 SPLUNK.KEY ( IT WORKS ) 

Then restart the splunk from GUI. Go to settings > server controls > Restart 😀

0 Karma

reswob4
Builder

The answer by @claudio.manig in this thread https://answers.splunk.com/answers/236495/splunk-kv-store-does-not-start.html helped me.

After verifying all the permissions discussed were set correctly on my server (linux), I stopped Splunk, deleted the mongod.lock file and restarted splunk and the error went away. I didn't need to run the repair utility.

Get Updates on the Splunk Community!

Unlock New Opportunities with Splunk Education: Explore Our Latest Courses!

At Splunk Education, we’re dedicated to providing top-tier learning experiences that cater to every skill ...

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...