topic Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error? in Knowledge Management

How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

gmckean — Thu, 19 Jan 2017 22:38:35 GMT

Issues with mongod startup

Failed to start KV Store process. See mongod.log and splunkd.log for details. 1/19/2017, 2:52:02 PM 
KV Store changed status to failed. KVStore process terminated. 1/19/2017, 2:52:01 PM    
KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details. 1/19/2017, 2:52:01 PM

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

coltwanger — Thu, 19 Jan 2017 23:35:53 GMT

What does mongod.log say about it?

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

hunters_splunk — Thu, 19 Jan 2017 23:42:16 GMT

Hi gmckean,

This might be caused by the incorrect mongo db key permission. Please try changing it to 400 using this command on Linux:

chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key

Hope this helps. Thanks!
Hunter

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

gmckean — Mon, 23 Jan 2017 15:48:47 GMT

mongod.log implied there was an issue with _tmp permissions. I corrected this then found other permission issues in other areas.

The corrective action was to chown splunk:splunk everything under ..../splunk_indexes/kvstore/mongo.

Some of the files here were owned by root:root.

Thanks for your help.

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

gmckean — Mon, 23 Jan 2017 15:53:30 GMT

The splunk.key file had 400 permission - other files in the splunk/kvstore/mongo dir were owned by root:root.
Making these splunk:splunk ownership fixed the problem.

Thanks

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

Abdul_hafeez — Mon, 08 May 2017 18:44:06 GMT

Thanks Hunters, it worked for me. I do not see that error anymore.

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

alisterwhipp — Thu, 07 Sep 2017 06:58:55 GMT

Permissions on a new install seem to be 600 these days. Maybe something's changed. Also, what's with the -R? just;

chmod 600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

should be fine. No need to recursively delve into directories of that file.

Add on a

chown splunk: /opt/splunk/var/lib/splunk/kvstore/mongo/*

to cover off the other issues about ownership noted in the thread.

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

Sp1unkin_it — Thu, 05 Oct 2017 20:19:12 GMT

Resolved the same issue for us. Thank you!

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

damode — Mon, 30 Oct 2017 23:35:48 GMT

Hi @alisterwhipp ,

how to do this on Windows ?
I have Splunk on Windows and I checked the file permission of Splunk.key, its already set to full control, still for my domain splunk account, I am getting the KVstore errors.

I started getting these errors after changing Splunk from Local account to domain account.
PLease help

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

gmckean — Tue, 31 Oct 2017 12:12:54 GMT

Sorry - no knowledge of the Windows side of things .

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

wsanderstii — Tue, 29 Sep 2020 17:45:36 GMT

What version? I don't see the kvstore directory in 6.6.5:

ls $SPLUNK_HOME/var/lib/splunk/kvstore

ls: cannot access
/opt/splunk/var/lib/splunk/kvstore: No
such file or directory

ls !$

ls $SPLUNK_HOME/var/lib/splunk

_audit.dat pgsql.dat srvadmin.dat
_blocksignature.dat resque.dat summary.dat database.dat

sandbox-dns-temp.dat task_admin.dat
dpr_worker.dat

sandbox-generic.dat _telemetry.dat
history.dat sandbox-ltm.dat

_thefishbucket.dat
_internal.dat seu-dpr.dat weblogs.dat
_introspection.dat seulogs.dat main.dat

seu_urlrequester.dat

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

wsanderstii — Fri, 19 Jan 2018 19:15:21 GMT

I downvoted this post because no such file in 6.6.5

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

coltwanger — Tue, 29 Sep 2020 17:41:41 GMT

That's pretty strange. However, the default path for kvstore is $SPLUNK_DB/kvstore. Perhaps you can verify a couple of things.

Where the KV Store is configured to reside:

/opt/splunk/bin/splunk btool server list | grep kvstore
If it returns dbPath=$SPLUNK_DB/kvstore, then verify your $SPLUNK_DB path is truly var/lib/splunk:

cat /opt/splunk/etc/splunk-launch.conf | grep SPLUNK_DB

Hopefully this points you in the right direction. I am on 7.0.1 and have not modified my default SPLUNK_DB path, and I am seeing /opt/splunk/var/lib/splunk/kvstore/ on both Windows and Linux installations. You aren't on Splunk Cloud by chance, are you?

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

millarma — Tue, 29 Sep 2020 18:16:21 GMT

I In Windows I went to C:\Program Files\Splunk\var\lib\splunk\modinputs\server\splunk_app_db_connect and gave write permission to the service account. I opened the folder properties and went to the security tab, selected my service account and gave it write permission.

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

gmckean — Tue, 27 Feb 2018 23:09:13 GMT

/opt/splunk/bin/splunk btool server list | grep kvstore
[introspection:generator:kvstore]
[kvstore]
dbPath = $SPLUNK_DB/kvstore

It was under splunk_indexes/kvstore/mongo/splunk.key - missing _indexes in original post

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

gmckean — Tue, 27 Feb 2018 23:11:56 GMT

/opt/splunk/bin/splunk btool server list | grep kvstore
[introspection:generator:kvstore]
[kvstore]
dbPath = $SPLUNK_DB/kvstore

..../splunk_indexes/kvstore/mongo

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

saisrujan28 — Tue, 20 Mar 2018 18:16:01 GMT

kvstore folder is missing in one of the search head in cluster can anyone explain why this happened and the search head which is missing with kvstore folder is kvstore captain.

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

omprakash9998 — Fri, 27 Apr 2018 18:05:10 GMT

I have the same error. Tried everything in this thread still the same issue.

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

coltwanger — Mon, 30 Apr 2018 14:29:48 GMT

What (if any) errors are there in mongod.log? Have you changed a SSL cert recently?

Re: How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

nawazns5038 — Fri, 19 Oct 2018 19:55:32 GMT

chmod: cannot access ‘/var/lib/splunk/kvstore/mongo/splunk.key’: No such file or directory