Knowledge Management

Why are we getting these errors: KV Store Process Terminated

rajindurbal
Path Finder

I have gotten 3 error on the search head. The errors are:

  • Failed to start KV Store process. See mongod.log and splunkd.log for details.
  • KV Store changed status to failed. KVStore process terminated.
  • KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

The problem only occurs on the search head, but the indexers are fine. This is a windows system. When I restart the search I get an error for "Cannot access appserver directly with appServerPorts configured." After a few minutes, splunk starts "normally". Not sure if the two issues are related. Could really use some help.

Labels (1)
Tags (1)
1 Solution

rajindurbal
Path Finder

So I worked with Splunk Support and what I had to do for this error was to:

  • Stop Splunk
  • rename the current mongo folder to old
  • Start Splunk
  • And you will see a new mongo folder created with all the components.

View solution in original post

morethanyell
Builder

None of the previous mentioned solutions worked for me. It turns out server.pem has expired from my machine so renewing locally fixed the issue. 

Renew how-to: Solved: Renewing server.pem certificate - Splunk Community

jbradshaw
Engager

I saw the message below on a cluster master used in a multisite environment.

 

Search peer s1-indexer04 has the following message: KV Store changed status to failed. KVStore process terminated

 

The following steps worked for me:

  • Stop splunk indexer 
    • Sample command: /opt/splunk/bin/splunk stop
  • remove the mongod.lock file 
    • Sample command: sudo rm -rf /data1/kvstore/mongo/mongod.lock
  • Starting Splunk
    • Sample command: /opt/splunk/bin/splunk start

Quintin_77
Engager

This worked for me. Thank you

0 Karma

nithishyk
Explorer

I have faced similar issue like this:

KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

I have fixed this by cleaning up the kvstore the particular search head which has the issue.

Stopped splunk.

splunk clean kvstore --local command.

start splunk.

Check status of kvstore.

woodcock
Esteemed Legend

Be advised that this command "cleans" by 'DESTROYING` the KVstore and reinitializing from scratch!

0 Karma

rajindurbal
Path Finder

So I worked with Splunk Support and what I had to do for this error was to:

  • Stop Splunk
  • rename the current mongo folder to old
  • Start Splunk
  • And you will see a new mongo folder created with all the components.

Spinner79
Explorer

@rajindurbal wrote:

So I worked with Splunk Support and what I had to do for this error was to:

  • Stop Splunk
  • rename the current mongo folder to old
  • Start Splunk
  • And you will see a new mongo folder created with all the components.

Hi,

I have done renaming the Mongo and recreate new, but still no luck.

Thank you

vinod743374
Communicator

I am Also facing the same issue,

if we do the Procedure you said in the solution will the KV store data is cleared or it will be the same.

can you please confirm once.

baonguyen78
New Member

I tried this on a distributed splunk setup, and upon restart, mongo.old got removed, kvstore error persists, and mongod isn't running.

0 Karma

scannon4
SplunkTrust
SplunkTrust

Where is the mongo folder located?

0 Karma

orezaie
Engager

$Splunk_DB/kvstore/mongo

$SPLUNK_DB , by default, is located in $SPLUNK_HOME/var/lib/splunk

0 Karma

CarsonZa
Contributor

@scannon4 $SPLUNK_HOME\var\lib\splunk\kvstore

daymauler
Explorer

It worked!!! Thanks

0 Karma

Spinner79
Explorer

did you re- generate the mongo?

Tags (1)
0 Karma

woodcock
Esteemed Legend

Be advised that this approach means that you will be reinitializing from scratch and you will lose ALL KVStore data (you do have a copy of it in old) unless you are in a cluster and you are only doing this on one Search Head!

0 Karma

ohignett
New Member

Does this mean reconfiguration of apps would be imminent?

0 Karma

CarsonZa
Contributor

@ohignett if they use the kvstore yes, for example Stream. If you were to clean the kvstore you would lose all configurations for that app. In my experience very little apps use the kvstore in this manner.

0 Karma

vadivel_parames
Explorer

Great Stuff!

0 Karma

nick405060
Motivator

wowwwwwwwwwwwwwwwwwwwwwwwwwww

0 Karma

jdthiele
Engager

This is what I needed to do after rsyncing the entire /opt/splunk folder over to a new file system to move splunk off of the root file system. Thanks for the help!!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...