Deployment Architecture

Failed to start KV store process : mongo could not be opened due to DBException 0

Contributor

After updating to 7.0.2 KV store is not starting. Looking in the mongod log this appears to the the error:

2018-03-22T23:54:15.194Z W STORAGE [initandlisten] database /opt/splunk/var/lib/splunk/kvstore/mongo s_searchNVnXrM8ANglxlhyhiYmtwGFA_dashbo2XgNZy78snDvL41QBsmrU3TU could not be opened due to DBException 0: assertion src/mongo/db/storage/mmap_v1/extent.h:81

I've tried changing permissions to 400
The mongo directory wasn't allowing me access to change the containing files to 400 so I changed that directory to 777.

0 Karma
1 Solution

Contributor

Solved this by deleting the database referenced in the error. Somehow it had gotten corrupted. Navigated to the mongo folder using the terminal, removed the database, stopped and started splunk.

Ran my output lookups to populate the KV store and the data came back.

After a couple minutes got this error:
2018-03-30T23:03:16.898Z I CONTROL [DataFileSync]
2018-03-30T23:03:16.892Z I - [DataFileSync] Fatal Assertion 17346
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] shutting down immediately to avoid corruption
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] error syncing data to disk, probably a disk error

Thinking there could disk errors I ran the OS badblocks disk check and found no errors.
Restarted machine and splunk came back up with no KV store errors. The KV store has been running for 24 hours with no errors.

View solution in original post

Contributor

Solved this by deleting the database referenced in the error. Somehow it had gotten corrupted. Navigated to the mongo folder using the terminal, removed the database, stopped and started splunk.

Ran my output lookups to populate the KV store and the data came back.

After a couple minutes got this error:
2018-03-30T23:03:16.898Z I CONTROL [DataFileSync]
2018-03-30T23:03:16.892Z I - [DataFileSync] Fatal Assertion 17346
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] shutting down immediately to avoid corruption
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] error syncing data to disk, probably a disk error

Thinking there could disk errors I ran the OS badblocks disk check and found no errors.
Restarted machine and splunk came back up with no KV store errors. The KV store has been running for 24 hours with no errors.

View solution in original post

Contributor

Just updated to maintenance version Splunk 7.0.3. KV store still not starting with the same error.

Champion

Hi can you try:

chmod 600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

And then:

./splunk show kvstore-status

Contributor

Thanks for the suggestion. Changed permissions on the .key file to 600. KV store still not starting with the same error.

0 Karma

Explorer

I am getting kV store related error in my Splunk

-Failed to start KV Store process. See mongod.log and splunkd.log for details.

-KV Store changed status to failed. KVStore process terminated..

-KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details

When I check the status "./splunk show kvstore-status" , It says "Failed"

I have checked below file permission "/opt/splunk/volr/splunk/kvstore/mongo/splunk.key" which is 600

And the DB location in "splunk-launch.conf" says-

Location where we want to store the indexed data.

SPLUNK_DB=/volr/splunk

Is there any help I can get here to troubleshoot it?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!