Failed to start KV store process : mongo could not be opened due to DBException 0

[cpt12tech]

After updating to 7.0.2 KV store is not starting. Looking in the mongod log this appears to the the error:

2018-03-22T23:54:15.194Z W STORAGE [initandlisten] database /opt/splunk/var/lib/splunk/kvstore/mongo s_searchNVnXrM8ANglxlhyhiYmtwGFA_dashbo2XgNZy78snDvL41QBsmrU3TU could not be opened due to DBException 0: assertion src/mongo/db/storage/mmap_v1/extent.h:81

I've tried changing permissions to 400
The mongo directory wasn't allowing me access to change the containing files to 400 so I changed that directory to 777.

[cpt12tech · Accepted solution]

Solved this by deleting the database referenced in the error. Somehow it had gotten corrupted. Navigated to the mongo folder using the terminal, removed the database, stopped and started splunk.

Ran my output lookups to populate the KV store and the data came back.

After a couple minutes got this error:
2018-03-30T23:03:16.898Z I CONTROL [DataFileSync]
2018-03-30T23:03:16.892Z I - [DataFileSync] Fatal Assertion 17346
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] shutting down immediately to avoid corruption
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] error syncing data to disk, probably a disk error

Thinking there could disk errors I ran the OS badblocks disk check and found no errors.
Restarted machine and splunk came back up with no KV store errors. The KV store has been running for 24 hours with no errors.

[cpt12tech]

Just updated to maintenance version Splunk 7.0.3. KV store still not starting with the same error.

[p_gurav]

Hi can you try:

chmod 600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

And then:

./splunk show kvstore-status

[cpt12tech]

Thanks for the suggestion. Changed permissions on the .key file to 600. KV store still not starting with the same error.

[rajuljain2605]

I am getting kV store related error in my Splunk

-Failed to start KV Store process. See mongod.log and splunkd.log for details.

-KV Store changed status to failed. KVStore process terminated..

-KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details

When I check the status "./splunk show kvstore-status" , It says "Failed"

I have checked below file permission "/opt/splunk/volr/splunk/kvstore/mongo/splunk.key" which is 600

And the DB location in "splunk-launch.conf" says-

Location where we want to store the indexed data.

SPLUNK_DB=/volr/splunk

Is there any help I can get here to troubleshoot it?