topic Re: KV Store Process Terminated in Knowledge Management

Why are we getting these errors: KV Store Process Terminated

rajindurbal — Wed, 22 Jun 2022 15:10:39 GMT

I have gotten 3 error on the search head. The errors are:

Failed to start KV Store process. See mongod.log and splunkd.log for details.
KV Store changed status to failed. KVStore process terminated.
KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

The problem only occurs on the search head, but the indexers are fine. This is a windows system. When I restart the search I get an error for "Cannot access appserver directly with appServerPorts configured." After a few minutes, splunk starts "normally". Not sure if the two issues are related. Could really use some help.

Re: KV Store Process Terminated

CarsonZa — Fri, 20 Jul 2018 21:05:55 GMT

have you looked at your mongod.log for any errors?

Re: KV Store Process Terminated

rajindurbal — Fri, 20 Jul 2018 21:58:10 GMT

The mongod.log says "Detected unclean shutdown - \mongod.lock is not empty. ".

Re: KV Store Process Terminated

CarsonZa — Sat, 21 Jul 2018 01:07:32 GMT

stop splunkd

delete this file
$SPLUNK_HOME\var\lib\splunk\kvstore\mongo\mongod.lock"

start splunkd

Re: KV Store Process Terminated

rajindurbal — Sat, 21 Jul 2018 01:22:16 GMT

Unfortunately, I have already tried this method. The issue continues to persist. After restarting, it starts up again with a new mongod.lock. I appreciate the feedback though.

Re: KV Store Process Terminated

CarsonZa — Sat, 21 Jul 2018 01:32:35 GMT

the lock file will always get remade. So the fact that it comes back is normal. But if you are still experiencing issues with the kvstore staying up then try to clean it using splunk clean kvstore --local

Re: KV Store Process Terminated

rajindurbal — Mon, 20 Aug 2018 23:06:35 GMT

So it turns out that the problem was for some reason, the mongo and kvstore folders did not have the right permissions. Therefore, splunk could not access them. After changing the permission and rebooting, the problems were resolved.

Re: KV Store Process Terminated

rajindurbal — Fri, 25 Jan 2019 15:59:31 GMT

So I worked with Splunk Support and what I had to do for this error was to:

Stop Splunk
rename the current mongo folder to old
Start Splunk
And you will see a new mongo folder created with all the components.

Re: KV Store Process Terminated

nithishyk — Wed, 20 Feb 2019 00:44:27 GMT

I have faced similar issue like this:

KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

I have fixed this by cleaning up the kvstore the particular search head which has the issue.

Stopped splunk.

splunk clean kvstore --local command.

start splunk.

Check status of kvstore.

Re: KV Store Process Terminated

jdthiele — Thu, 09 May 2019 18:47:55 GMT

This is what I needed to do after rsyncing the entire /opt/splunk folder over to a new file system to move splunk off of the root file system. Thanks for the help!!

Re: KV Store Process Terminated

nick405060 — Tue, 21 May 2019 22:30:15 GMT

wowwwwwwwwwwwwwwwwwwwwwwwwwww

Re: KV Store Process Terminated

vadivel_parames — Tue, 19 Nov 2019 12:11:08 GMT

Great Stuff!

Re: KV Store Process Terminated

woodcock — Fri, 22 Nov 2019 17:46:58 GMT

What are the correct permissions, exactly?

Re: KV Store Process Terminated

woodcock — Fri, 22 Nov 2019 17:51:36 GMT

Be advised that this command "cleans" by 'DESTROYING` the KVstore and reinitializing from scratch!

Re: KV Store Process Terminated

woodcock — Fri, 22 Nov 2019 17:53:02 GMT

Be advised that this approach means that you will be reinitializing from scratch and you will lose ALL KVStore data (you do have a copy of it in old) unless you are in a cluster and you are only doing this on one Search Head!

Re: KV Store Process Terminated

mhouse — Tue, 03 Dec 2019 15:15:49 GMT

What were the permissions before you made the change and what permissions settings did you change it to in order to get it to work?

Re: KV Store Process Terminated

daymauler — Fri, 10 Jan 2020 18:18:05 GMT

It worked!!! Thanks

Re: KV Store Process Terminated

scannon4 — Fri, 24 Jan 2020 13:31:27 GMT

Where is the mongo folder located?

Re: KV Store Process Terminated

CarsonZa — Fri, 24 Jan 2020 15:49:22 GMT

@scannon4 $SPLUNK_HOME\var\lib\splunk\kvstore

Re: KV Store Process Terminated

ohignett — Thu, 30 Jan 2020 17:55:58 GMT

Does this mean reconfiguration of apps would be imminent?