Deployment Architecture

Failed to start KV store process : mongo could not be opened due to DBException 0

cpt12tech
Contributor

After updating to 7.0.2 KV store is not starting. Looking in the mongod log this appears to the the error:

2018-03-22T23:54:15.194Z W STORAGE [initandlisten] database /opt/splunk/var/lib/splunk/kvstore/mongo s_searchNVnXrM8ANglxlhyhiYmtwGFA_dashbo2XgNZy78snDvL41QBsmrU3TU could not be opened due to DBException 0: assertion src/mongo/db/storage/mmap_v1/extent.h:81

I've tried changing permissions to 400
The mongo directory wasn't allowing me access to change the containing files to 400 so I changed that directory to 777.

0 Karma
1 Solution

cpt12tech
Contributor

Solved this by deleting the database referenced in the error. Somehow it had gotten corrupted. Navigated to the mongo folder using the terminal, removed the database, stopped and started splunk.

Ran my output lookups to populate the KV store and the data came back.

After a couple minutes got this error:
2018-03-30T23:03:16.898Z I CONTROL [DataFileSync]
2018-03-30T23:03:16.892Z I - [DataFileSync] Fatal Assertion 17346
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] shutting down immediately to avoid corruption
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] error syncing data to disk, probably a disk error

Thinking there could disk errors I ran the OS badblocks disk check and found no errors.
Restarted machine and splunk came back up with no KV store errors. The KV store has been running for 24 hours with no errors.

View solution in original post

cpt12tech
Contributor

Solved this by deleting the database referenced in the error. Somehow it had gotten corrupted. Navigated to the mongo folder using the terminal, removed the database, stopped and started splunk.

Ran my output lookups to populate the KV store and the data came back.

After a couple minutes got this error:
2018-03-30T23:03:16.898Z I CONTROL [DataFileSync]
2018-03-30T23:03:16.892Z I - [DataFileSync] Fatal Assertion 17346
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] shutting down immediately to avoid corruption
2018-03-30T23:03:16.892Z I CONTROL [DataFileSync] error syncing data to disk, probably a disk error

Thinking there could disk errors I ran the OS badblocks disk check and found no errors.
Restarted machine and splunk came back up with no KV store errors. The KV store has been running for 24 hours with no errors.

cpt12tech
Contributor

Just updated to maintenance version Splunk 7.0.3. KV store still not starting with the same error.

p_gurav
Champion

Hi can you try:

chmod 600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

And then:

./splunk show kvstore-status

cpt12tech
Contributor

Thanks for the suggestion. Changed permissions on the .key file to 600. KV store still not starting with the same error.

0 Karma

rajuljain2605
Explorer

I am getting kV store related error in my Splunk

-Failed to start KV Store process. See mongod.log and splunkd.log for details.

-KV Store changed status to failed. KVStore process terminated..

-KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details

When I check the status "./splunk show kvstore-status" , It says "Failed"

I have checked below file permission "/opt/splunk/volr/splunk/kvstore/mongo/splunk.key" which is 600

And the DB location in "splunk-launch.conf" says-

Location where we want to store the indexed data.

SPLUNK_DB=/volr/splunk

Is there any help I can get here to troubleshoot it?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...