Knowledge Management

KV Store Process Terminated

Path Finder

I have gotten 3 error on the search head. The errors are:

  • Failed to start KV Store process. See mongod.log and splunkd.log for details.
  • KV Store changed status to failed. KVStore process terminated.
  • KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

The problem only occurs on the search head, but the indexers are fine. This is a windows system. When I restart the search I get an error for "Cannot access appserver directly with appServerPorts configured." After a few minutes, splunk starts "normally". Not sure if the two issues are related. Could really use some help.

Labels (1)
Tags (1)
1 Solution

Path Finder

So I worked with Splunk Support and what I had to do for this error was to:

  • Stop Splunk
  • rename the current mongo folder to old
  • Start Splunk
  • And you will see a new mongo folder created with all the components.

View solution in original post

New Member

I have faced similar issue like this:

KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

I have fixed this by cleaning up the kvstore the particular search head which has the issue.

Stopped splunk.

splunk clean kvstore --local command.

start splunk.

Check status of kvstore.

0 Karma

Esteemed Legend

Be advised that this command "cleans" by 'DESTROYING` the KVstore and reinitializing from scratch!

0 Karma

Path Finder

So I worked with Splunk Support and what I had to do for this error was to:

  • Stop Splunk
  • rename the current mongo folder to old
  • Start Splunk
  • And you will see a new mongo folder created with all the components.

View solution in original post

New Member

I tried this on a distributed splunk setup, and upon restart, mongo.old got removed, kvstore error persists, and mongod isn't running.

0 Karma

SplunkTrust
SplunkTrust

Where is the mongo folder located?

0 Karma

Contributor

@scannon4 $SPLUNK_HOME\var\lib\splunk\kvstore

0 Karma

Explorer

It worked!!! Thanks

0 Karma

Esteemed Legend

Be advised that this approach means that you will be reinitializing from scratch and you will lose ALL KVStore data (you do have a copy of it in old) unless you are in a cluster and you are only doing this on one Search Head!

0 Karma

New Member

Does this mean reconfiguration of apps would be imminent?

0 Karma

Contributor

@ohignett if they use the kvstore yes, for example Stream. If you were to clean the kvstore you would lose all configurations for that app. In my experience very little apps use the kvstore in this manner.

0 Karma

Great Stuff!

0 Karma

Motivator

wowwwwwwwwwwwwwwwwwwwwwwwwwww

0 Karma

Engager

This is what I needed to do after rsyncing the entire /opt/splunk folder over to a new file system to move splunk off of the root file system. Thanks for the help!!

0 Karma

Path Finder

So it turns out that the problem was for some reason, the mongo and kvstore folders did not have the right permissions. Therefore, splunk could not access them. After changing the permission and rebooting, the problems were resolved.

0 Karma

New Member

What were the permissions before you made the change and what permissions settings did you change it to in order to get it to work?

0 Karma

Esteemed Legend

What are the correct permissions, exactly?

0 Karma

Contributor

stop splunkd

delete this file
$SPLUNK_HOME\var\lib\splunk\kvstore\mongo\mongod.lock"

start splunkd

0 Karma

Path Finder

Unfortunately, I have already tried this method. The issue continues to persist. After restarting, it starts up again with a new mongod.lock. I appreciate the feedback though.

0 Karma

Contributor

the lock file will always get remade. So the fact that it comes back is normal. But if you are still experiencing issues with the kvstore staying up then try to clean it using splunk clean kvstore --local

0 Karma

Contributor

have you looked at your mongod.log for any errors?

0 Karma