In Splunk Enterprise Security, will you help me fix the following error: "Failed to start KV Store process. See mongod.log and splunkd.log for details."

[christianubeda]

Hi team!

I need help.

I have these errors from a long time ago but I didn't notice. Everything works but I need to fix these errors.

My version is 6.1.1.

Thank you!

[LukeMurphey]

I recommend trying the following solutions to see if it fixes your problem:

• https://answers.splunk.com/answers/490134/how-to-resolve-issues-with-mongod-startup-such-as.html
• https://answers.splunk.com/answers/397029/kv-store-lookup-failing-with-error-about-kv-store.html
• https://answers.splunk.com/answers/673243/kv-store-process-terminated.html
• https://answers.splunk.com/answers/690118/why-is-the-kv-store-process-failing-after-upgrade.html

I had the same error in the past and in my case it turned out to be due to the permissions on the splunk.key file (the first link above).

You can also check the following page for more troubleshooting tips: http://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/TroubleshootKVstore

[christianubeda]

It didn´t works.

I got this from mongod.log

2018-11-06T20:06:02.952Z I STORAGE [initandlisten]
2018-11-06T20:06:02.952Z I STORAGE [initandlisten] ** WARNING: Readahead for /opt/splunk/var/lib/splunk/kvstore/mongo is set to 4096KB
2018-11-06T20:06:02.952Z I STORAGE [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2018-11-06T20:06:02.952Z I STORAGE [initandlisten] ** http://dochub.mongodb.org/core/readahead
2018-11-06T20:06:02.953Z I JOURNAL [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal
2018-11-06T20:06:02.953Z I JOURNAL [initandlisten] recover : no journal files present, no recovery needed
2018-11-06T20:07:17.756Z I JOURNAL [durability] Durability thread started
2018-11-06T20:07:17.756Z I JOURNAL [journal writer] Journal writer thread started
2018-11-06T20:07:31.258Z I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-11-06T20:07:31.258Z I CONTROL [initandlisten]
2018-11-06T20:07:31.260Z I CONTROL [initandlisten]
2018-11-06T20:07:31.260Z I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine.
2018-11-06T20:07:31.260Z I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems:
2018-11-06T20:07:31.260Z I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options]
2018-11-06T20:07:31.260Z I CONTROL [initandlisten]
2018-11-06T20:10:59.930Z I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2018-11-06T20:10:59.930Z W REPL [signalProcessingThread] ReplicationCoordinatorImpl::shutdown() called before startReplication() finished. Shutting down without cleaning up the replication system

Do you have any ideas?

[LukeMurphey]

I recommend opening a case with Support. There are many reasons why KV store might not start and I suspect Support will be able to get to the root cause faster than I can via Answers.