Knowledge Management

Why are we getting these errors: KV Store Process Terminated

rajindurbal
Path Finder

I have gotten 3 error on the search head. The errors are:

  • Failed to start KV Store process. See mongod.log and splunkd.log for details.
  • KV Store changed status to failed. KVStore process terminated.
  • KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

The problem only occurs on the search head, but the indexers are fine. This is a windows system. When I restart the search I get an error for "Cannot access appserver directly with appServerPorts configured." After a few minutes, splunk starts "normally". Not sure if the two issues are related. Could really use some help.

Labels (1)
Tags (1)
1 Solution

rajindurbal
Path Finder

So I worked with Splunk Support and what I had to do for this error was to:

  • Stop Splunk
  • rename the current mongo folder to old
  • Start Splunk
  • And you will see a new mongo folder created with all the components.

View solution in original post

morethanyell
Builder

None of the previous mentioned solutions worked for me. It turns out server.pem has expired from my machine so renewing locally fixed the issue. 

Renew how-to: Solved: Renewing server.pem certificate - Splunk Community

jbradshaw
Engager

I saw the message below on a cluster master used in a multisite environment.

 

Search peer s1-indexer04 has the following message: KV Store changed status to failed. KVStore process terminated

 

The following steps worked for me:

  • Stop splunk indexer 
    • Sample command: /opt/splunk/bin/splunk stop
  • remove the mongod.lock file 
    • Sample command: sudo rm -rf /data1/kvstore/mongo/mongod.lock
  • Starting Splunk
    • Sample command: /opt/splunk/bin/splunk start

Quintin_77
Engager

This worked for me. Thank you

0 Karma

nithishyk
Explorer

I have faced similar issue like this:

KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

I have fixed this by cleaning up the kvstore the particular search head which has the issue.

Stopped splunk.

splunk clean kvstore --local command.

start splunk.

Check status of kvstore.

woodcock
Esteemed Legend

Be advised that this command "cleans" by 'DESTROYING` the KVstore and reinitializing from scratch!

0 Karma

rajindurbal
Path Finder

So I worked with Splunk Support and what I had to do for this error was to:

  • Stop Splunk
  • rename the current mongo folder to old
  • Start Splunk
  • And you will see a new mongo folder created with all the components.

vinod743374
Communicator

I am Also facing the same issue,

if we do the Procedure you said in the solution will the KV store data is cleared or it will be the same.

can you please confirm once.

0 Karma

baonguyen78
New Member

I tried this on a distributed splunk setup, and upon restart, mongo.old got removed, kvstore error persists, and mongod isn't running.

0 Karma

scannon4
SplunkTrust
SplunkTrust

Where is the mongo folder located?

0 Karma

orezaie
Engager

$Splunk_DB/kvstore/mongo

$SPLUNK_DB , by default, is located in $SPLUNK_HOME/var/lib/splunk

0 Karma

CarsonZa
Contributor

@scannon4 $SPLUNK_HOME\var\lib\splunk\kvstore

daymauler
Explorer

It worked!!! Thanks

0 Karma

woodcock
Esteemed Legend

Be advised that this approach means that you will be reinitializing from scratch and you will lose ALL KVStore data (you do have a copy of it in old) unless you are in a cluster and you are only doing this on one Search Head!

0 Karma

ohignett
New Member

Does this mean reconfiguration of apps would be imminent?

0 Karma

CarsonZa
Contributor

@ohignett if they use the kvstore yes, for example Stream. If you were to clean the kvstore you would lose all configurations for that app. In my experience very little apps use the kvstore in this manner.

0 Karma

vadivel_parames
Explorer

Great Stuff!

0 Karma

nick405060
Motivator

wowwwwwwwwwwwwwwwwwwwwwwwwwww

0 Karma

jdthiele
Engager

This is what I needed to do after rsyncing the entire /opt/splunk folder over to a new file system to move splunk off of the root file system. Thanks for the help!!

0 Karma

GregorGoetz
New Member

Had the same issue moving /opt from root fs to mounted /opt with larger partition after moving ./splunk folder to that new /opt.

0 Karma

rajindurbal
Path Finder

So it turns out that the problem was for some reason, the mongo and kvstore folders did not have the right permissions. Therefore, splunk could not access them. After changing the permission and rebooting, the problems were resolved.

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...