How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

[gmckean]

Issues with mongod startup

Failed to start KV Store process. See mongod.log and splunkd.log for details. 1/19/2017, 2:52:02 PM 
KV Store changed status to failed. KVStore process terminated. 1/19/2017, 2:52:01 PM    
KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details. 1/19/2017, 2:52:01 PM

[wsanderstii]

I downvoted this post because no such file in 6.6.5

[gmckean]

Sorry - no knowledge of the Windows side of things .

[omprakash9998]

I have the same error. Tried everything in this thread still the same issue.

[coltwanger]

What (if any) errors are there in mongod.log? Have you changed a SSL cert recently?

[millarma]

I In Windows I went to C:\Program Files\Splunk\var\lib\splunk\modinputs\server\splunk_app_db_connect and gave write permission to the service account. I opened the folder properties and went to the security tab, selected my service account and gave it write permission.

[gmckean]

The splunk.key file had 400 permission - other files in the splunk/kvstore/mongo dir were owned by root:root.
Making these splunk:splunk ownership fixed the problem.

Thanks

[Sp1unkin_it]

Resolved the same issue for us. Thank you!

[alisterwhipp]

Permissions on a new install seem to be 600 these days. Maybe something's changed. Also, what's with the -R? just;

chmod 600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

should be fine. No need to recursively delve into directories of that file.

Add on a

chown splunk: /opt/splunk/var/lib/splunk/kvstore/mongo/*

to cover off the other issues about ownership noted in the thread.

[damode]

Hi @alisterwhipp ,

how to do this on Windows ?
I have Splunk on Windows and I checked the file permission of Splunk.key, its already set to full control, still for my domain splunk account, I am getting the KVstore errors.

I started getting these errors after changing Splunk from Local account to domain account.
PLease help

[nioann]

@damodeI have the same issue in a Windows machine after changing Splunk from Local account to domain account. Did you find a solution?

[Abdul_hafeez]

Thanks Hunters, it worked for me. I do not see that error anymore.

[coltwanger]

What does mongod.log say about it?

[gmckean]

mongod.log implied there was an issue with _tmp permissions. I corrected this then found other permission issues in other areas.

The corrective action was to chown splunk:splunk everything under ..../splunk_indexes/kvstore/mongo.

Some of the files here were owned by root:root.

Thanks for your help.