topic Failed to start KV store process : mongo could not be opened due to DBException 0 in Deployment Architecture

Failed to start KV store process : mongo could not be opened due to DBException 0

cpt12tech — Tue, 29 Sep 2020 18:38:12 GMT

After updating to 7.0.2 KV store is not starting. Looking in the mongod log this appears to the the error:

2018-03-22T23:54:15.194Z W STORAGE [initandlisten] database /opt/splunk/var/lib/splunk/kvstore/mongo s_searchNVnXrM8ANglxlhyhiYmtwGFA_dashbo2XgNZy78snDvL41QBsmrU3TU could not be opened due to DBException 0: assertion src/mongo/db/storage/mmap_v1/extent.h:81

I've tried changing permissions to 400
The mongo directory wasn't allowing me access to change the containing files to 400 so I changed that directory to 777.

Re: Failed to start KV store process : mongo could not be opened due to DBException 0

p_gurav — Fri, 23 Mar 2018 03:18:47 GMT

Hi can you try:

chmod 600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

And then:

./splunk show kvstore-status

Re: Failed to start KV store process : mongo could not be opened due to DBException 0

cpt12tech — Sat, 24 Mar 2018 00:42:25 GMT

Thanks for the suggestion. Changed permissions on the .key file to 600. KV store still not starting with the same error.

Re: Failed to start KV store process : mongo could not be opened due to DBException 0

cpt12tech — Sat, 24 Mar 2018 00:43:00 GMT

Just updated to maintenance version Splunk 7.0.3. KV store still not starting with the same error.

Re: Failed to start KV store process : mongo could not be opened due to DBException 0

cpt12tech — Thu, 05 Apr 2018 17:39:17 GMT

Solved this by deleting the database referenced in the error. Somehow it had gotten corrupted. Navigated to the mongo folder using the terminal, removed the database, stopped and started splunk.

Ran my output lookups to populate the KV store and the data came back.

After a couple minutes got this error:
2018-03-30T23:03:16.898Z I CONTROL [DataFileSync]
2018-03-30T23:03:16.892Z I - [DataFileSync] Fatal Assertion 17346
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] shutting down immediately to avoid corruption
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] error syncing data to disk, probably a disk error

Thinking there could disk errors I ran the OS badblocks disk check and found no errors.
Restarted machine and splunk came back up with no KV store errors. The KV store has been running for 24 hours with no errors.

Re: Failed to start KV store process : mongo could not be opened due to DBException 0

rajuljain2605 — Thu, 20 Jun 2019 10:32:51 GMT

I am getting kV store related error in my Splunk

-Failed to start KV Store process. See mongod.log and splunkd.log for details.

-KV Store changed status to failed. KVStore process terminated..

-KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details

When I check the status "./splunk show kvstore-status" , It says "Failed"

I have checked below file permission "/opt/splunk/volr/splunk/kvstore/mongo/splunk.key" which is 600

And the DB location in "splunk-launch.conf" says-

Location where we want to store the indexed data.

SPLUNK_DB=/volr/splunk

Is there any help I can get here to troubleshoot it?