Knowledge Management

How to resolve issues with mongod startup such as "Failed to start KV Store process" error?

Explorer

Issues with mongod startup

Failed to start KV Store process. See mongod.log and splunkd.log for details. 1/19/2017, 2:52:02 PM 
KV Store changed status to failed. KVStore process terminated. 1/19/2017, 2:52:01 PM    
KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details. 1/19/2017, 2:52:01 PM
1 Solution

Splunk Employee
Splunk Employee

Hi gmckean,

This might be caused by the incorrect mongo db key permission. Please try changing it to 400 using this command on Linux:

chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key

Hope this helps. Thanks!
Hunter

View solution in original post

Explorer

My fault was that the certificate in server.pem no longer was valid.
Its only valid 3 years

See the thread:
https://answers.splunk.com/answers/457893/after-upgrading-to-650-kv-store-will-not-start.html

Old cert
openssl x509 -enddate -noout -in ./server.pem
notAfter=Mar 15 18:52:32 2020 GMT

New cert
openssl x509 -enddate -noout -in ./server.pem
notAfter=Mar 17 07:38:09 2023 GMT

0 Karma

Explorer

/opt/splunk/bin/splunk btool server list | grep kvstore
[introspection:generator:kvstore]
[kvstore]
dbPath = $SPLUNK_DB/kvstore

It was under splunk_indexes/kvstore/mongo/splunk.key - missing _indexes in original post

Path Finder

What version? I don't see the kvstore directory in 6.6.5:

ls $SPLUNK_HOME/var/lib/splunk/kvstore

ls: cannot access
/opt/splunk/var/lib/splunk/kvstore: No
such file or directory

ls !$

ls $SPLUNK_HOME/var/lib/splunk

audit.dat pgsql.dat srvadmin.dat
_blocksignature.dat resque.dat summary.dat database.dat

sandbox-dns-temp.dat task
admin.dat
dprworker.dat

sandbox-generic.dat _telemetry.dat
history.dat sandbox-ltm.dat

_thefishbucket.dat
_internal.dat seu-dpr.dat weblogs.dat
_introspection.dat seulogs.dat main.dat

seu
urlrequester.dat

Contributor

That's pretty strange. However, the default path for kvstore is $SPLUNK_DB/kvstore. Perhaps you can verify a couple of things.

  1. Where the KV Store is configured to reside:

    /opt/splunk/bin/splunk btool server list | grep kvstore

  2. If it returns dbPath=$SPLUNKDB/kvstore, then verify your $SPLUNKDB path is truly var/lib/splunk:

    cat /opt/splunk/etc/splunk-launch.conf | grep SPLUNK_DB

Hopefully this points you in the right direction. I am on 7.0.1 and have not modified my default SPLUNK_DB path, and I am seeing /opt/splunk/var/lib/splunk/kvstore/ on both Windows and Linux installations. You aren't on Splunk Cloud by chance, are you?

0 Karma

Explorer

/opt/splunk/bin/splunk btool server list | grep kvstore
[introspection:generator:kvstore]
[kvstore]
dbPath = $SPLUNK_DB/kvstore

..../splunk_indexes/kvstore/mongo

0 Karma

Splunk Employee
Splunk Employee

Hi gmckean,

This might be caused by the incorrect mongo db key permission. Please try changing it to 400 using this command on Linux:

chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key

Hope this helps. Thanks!
Hunter

View solution in original post

Explorer

I've tried this method, as well as the link below but without luck
https://answers.splunk.com/answers/655729/kv-store-errors-kv-store-changed-status-to-failed-1.html

Any help would be eternally appreciated ...

0 Karma

Engager

We had the same issue but we validated the permissions and changed it to 400
it RESOLVED our issue.

Many Thanks!!! Hunter

0 Karma

Builder

chmod: cannot access ‘/var/lib/splunk/kvstore/mongo/splunk.key’: No such file or directory

0 Karma

Splunk Employee
Splunk Employee

For Windows, if you face this error, you need to take ownership of the kvstore folder, and assign Full Control to System and Administrators (they most likely have "Special Permissions").

Most likely this is: C:\Program Files\Splunk\var\lib\splunk\kvstore\

The problem isn't that the folder is Read Only. Rather, because your folder was created on a different installation of Windows you no longer have NTFS security permissions to access (read) the folder.

Correct this by following these steps to take ownership and then grant yourself full access to the folder.

1.) Right-click the folder > **Properties**
2.) *Security* tab > **Advanced**
3.) Click **Change** to the right of *Owner*
4.) Enter `Users` into box and click **OK**
5.) Enable the checkbox **Replace owner on subcontainers and objects** then click **Apply**
6.) If prompted that *You do not have permissions to read...* click **Yes**
7.) Completely close out of the *Advanced Security Settings* dialog
8.) Right-click the folder > **Properties**
9.) *Security* tab > **Edit...**
10.) **Add...**
11.) Enter `Users` into box and click **OK**
12.) Enable the **Full Control** checkbox then click **OK**

via Twisty Impersonator at SuperUser.com : https://superuser.com/questions/846143/remove-read-only-attribute-from-folder-after-windows-reinstal...

I did this and the error went away on my Win machine and a customer's. This may happen because of certain major Windows Updates as well.

0 Karma

Builder

how about linux ?

0 Karma

Splunk Employee
Splunk Employee

Did you see @gmckean comment above?

https://answers.splunk.com/answers/490134/how-to-resolve-issues-with-mongod-startup-such-as.html#com...

Try doing chmod on the entire folder instead of just a specific file. Before doing so, you can check if the splunk user has access to all the files/subfolders or if they're owned by root. The "splunk" user owning/running splunk should have access to this location and all files.

0 Karma

Explorer

kvstore folder is missing in one of the search head in cluster can anyone explain why this happened and the search head which is missing with kvstore folder is kvstore captain.

0 Karma

Path Finder

I downvoted this post because no such file in 6.6.5

0 Karma

Explorer

Sorry - no knowledge of the Windows side of things .

0 Karma

Path Finder

I have the same error. Tried everything in this thread still the same issue.

0 Karma

Contributor

What (if any) errors are there in mongod.log? Have you changed a SSL cert recently?

0 Karma

Explorer

I In Windows I went to C:\Program Files\Splunk\var\lib\splunk\modinputs\server\splunkappdb_connect and gave write permission to the service account. I opened the folder properties and went to the security tab, selected my service account and gave it write permission.

0 Karma

Explorer

The splunk.key file had 400 permission - other files in the splunk/kvstore/mongo dir were owned by root:root.
Making these splunk:splunk ownership fixed the problem.

Thanks

0 Karma

Explorer

Resolved the same issue for us. Thank you!

0 Karma