How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

[gmckean]

Issues with mongod startup

Failed to start KV Store process. See mongod.log and splunkd.log for details. 1/19/2017, 2:52:02 PM 
KV Store changed status to failed. KVStore process terminated. 1/19/2017, 2:52:01 PM    
KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details. 1/19/2017, 2:52:01 PM

[jotne]

My fault was that the certificate in server.pem no longer was valid.
Its only valid 3 years

See the thread:
https://answers.splunk.com/answers/457893/after-upgrading-to-650-kv-store-will-not-start.html

Old cert
openssl x509 -enddate -noout -in ./server.pem
notAfter=Mar 15 18:52:32 2020 GMT

New cert
openssl x509 -enddate -noout -in ./server.pem
notAfter=Mar 17 07:38:09 2023 GMT

[duartet]

The exact same thing happened to me.

Was updating an indexer from 7.3.9 to 8.1.6, using the new parameter on kvstore stanza:

[kvstore]
storageEngineMigration=true

But the kvstore wasn't migrating, giving this error:

Starting KV Store storage engine upgrade:
Phase 1 (dump) of 2:
ERROR: Failed to migrate to storage engine wiredTiger, reason=KVStore service will not start because kvstore process terminated

It was nothing more nothing less than the certificate that had expired 5 days ago!

Thanks for the tip!

[gmckean]

/opt/splunk/bin/splunk btool server list | grep kvstore
[introspection:generator:kvstore]
[kvstore]
dbPath = $SPLUNK_DB/kvstore

It was under splunk_indexes/kvstore/mongo/splunk.key - missing _indexes in original post

[wsanderstii]

What version? I don't see the kvstore directory in 6.6.5:

ls $SPLUNK_HOME/var/lib/splunk/kvstore

ls: cannot access
/opt/splunk/var/lib/splunk/kvstore: No
such file or directory

ls !$

ls $SPLUNK_HOME/var/lib/splunk

_audit.dat pgsql.dat srvadmin.dat
_blocksignature.dat resque.dat summary.dat database.dat

sandbox-dns-temp.dat task_admin.dat
dpr_worker.dat

sandbox-generic.dat _telemetry.dat
history.dat sandbox-ltm.dat

_thefishbucket.dat
_internal.dat seu-dpr.dat weblogs.dat
_introspection.dat seulogs.dat main.dat

seu_urlrequester.dat

[coltwanger]

That's pretty strange. However, the default path for kvstore is $SPLUNK_DB/kvstore. Perhaps you can verify a couple of things.

1. Where the KV Store is configured to reside:

   /opt/splunk/bin/splunk btool server list | grep kvstore

2. If it returns dbPath=$SPLUNK_DB/kvstore, then verify your $SPLUNK_DB path is truly var/lib/splunk:

   cat /opt/splunk/etc/splunk-launch.conf | grep SPLUNK_DB

Hopefully this points you in the right direction. I am on 7.0.1 and have not modified my default SPLUNK_DB path, and I am seeing /opt/splunk/var/lib/splunk/kvstore/ on both Windows and Linux installations. You aren't on Splunk Cloud by chance, are you?

[gmckean]

/opt/splunk/bin/splunk btool server list | grep kvstore
[introspection:generator:kvstore]
[kvstore]
dbPath = $SPLUNK_DB/kvstore

..../splunk_indexes/kvstore/mongo

[hunters_splunk · Accepted solution]

Hi gmckean,

This might be caused by the incorrect mongo db key permission. Please try changing it to 400 using this command on Linux:

chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key

Hope this helps. Thanks!
Hunter

[ChrisTheDude]

Changing the permissions on the splunk.key file to read only/400 did the trick for me. Thanks!!!

[Spinner79]

mine is windows no need to set the chmod right

[AR0C87446]

How to resolve in Windows as After changing Splunk from http to https getting Kv store error and could not load lookup=lookupsite error

what will be the cause and solution.

[digantopaul]

The logs in mongod.log show as below;

ACCESS [main] permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open

Thank you for this resolution

Solved: Re: How to resolve issues with mongod startup such... - Splunk Community

[peterkn]

I've tried this method, as well as the link below but without luck
https://answers.splunk.com/answers/655729/kv-store-errors-kv-store-changed-status-to-failed-1.html

Any help would be eternally appreciated ...

[rbalasa]

We had the same issue but we validated the permissions and changed it to 400
it RESOLVED our issue.

Many Thanks!!! Hunter

[nawazns5038]

chmod: cannot access ‘/var/lib/splunk/kvstore/mongo/splunk.key’: No such file or directory

[rkantamaneni_sp]

For Windows, if you face this error, you need to take ownership of the kvstore folder, and assign Full Control to System and Administrators (they most likely have "Special Permissions").

Most likely this is: C:\Program Files\Splunk\var\lib\splunk\kvstore\

The problem isn't that the folder is Read Only. Rather, because your folder was created on a different installation of Windows you no longer have NTFS security permissions to access (read) the folder.

Correct this by following these steps to take ownership and then grant yourself full access to the folder.

1.) Right-click the folder > **Properties**
2.) *Security* tab > **Advanced**
3.) Click **Change** to the right of *Owner*
4.) Enter `Users` into box and click **OK**
5.) Enable the checkbox **Replace owner on subcontainers and objects** then click **Apply**
6.) If prompted that *You do not have permissions to read...* click **Yes**
7.) Completely close out of the *Advanced Security Settings* dialog
8.) Right-click the folder > **Properties**
9.) *Security* tab > **Edit...**
10.) **Add...**
11.) Enter `Users` into box and click **OK**
12.) Enable the **Full Control** checkbox then click **OK**

---

via Twisty Impersonator at SuperUser.com : https://superuser.com/questions/846143/remove-read-only-attribute-from-folder-after-windows-reinstal...

I did this and the error went away on my Win machine and a customer's. This may happen because of certain major Windows Updates as well.

[Gregski11]

this was a no dice for me with the error:

 F - [main] Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609
 F - [main] \n\n***aborting after fassert() failure\n\n

[pavankumarh]

https://community.splunk.com/t5/Knowledge-Management/Why-is-KV-Store-certificate-renewal-not-working... 

On Windows, you may get the following error message in mongod.log:

Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609

To fix the error that causes mongod to terminate, you need the following in addition to deleting server.pem:

Open Windows certificate management MMC for the local computer ( certlm.msc )
Navigate to Personal > Certificates
Delete any entries named SplunkServerDefaultCert

Restart splunk. 

[nawazns5038]

how about linux ?

[rkantamaneni_sp]

Did you see @gmckean comment above?

https://answers.splunk.com/answers/490134/how-to-resolve-issues-with-mongod-startup-such-as.html#com...

Try doing chmod on the entire folder instead of just a specific file. Before doing so, you can check if the splunk user has access to all the files/subfolders or if they're owned by root. The "splunk" user owning/running splunk should have access to this location and all files.

[saisrujan28]

kvstore folder is missing in one of the search head in cluster can anyone explain why this happened and the search head which is missing with kvstore folder is kvstore captain.