Why are we getting these errors: KV Store Process Terminated

[rajindurbal]

I have gotten 3 error on the search head. The errors are:

• Failed to start KV Store process. See mongod.log and splunkd.log for details.
• KV Store changed status to failed. KVStore process terminated.
• KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

The problem only occurs on the search head, but the indexers are fine. This is a windows system. When I restart the search I get an error for "Cannot access appserver directly with appServerPorts configured." After a few minutes, splunk starts "normally". Not sure if the two issues are related. Could really use some help.

[morethanyell]

None of the previous mentioned solutions worked for me. It turns out server.pem has expired from my machine so renewing locally fixed the issue. 

Renew how-to: Solved: Renewing server.pem certificate - Splunk Community

[jbradshaw]

I saw the message below on a cluster master used in a multisite environment.

Search peer s1-indexer04 has the following message: KV Store changed status to failed. KVStore process terminated

The following steps worked for me:

• Stop splunk indexer 
  • Sample command: /opt/splunk/bin/splunk stop
• remove the mongod.lock file 
  • Sample command: sudo rm -rf /data1/kvstore/mongo/mongod.lock
• Starting Splunk
  • Sample command: /opt/splunk/bin/splunk start

[Quintin_77]

This worked for me. Thank you

[nithishyk]

I have faced similar issue like this:

KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

I have fixed this by cleaning up the kvstore the particular search head which has the issue.

Stopped splunk.

splunk clean kvstore --local command.

start splunk.

Check status of kvstore.

[woodcock]

Be advised that this command "cleans" by 'DESTROYING` the KVstore and reinitializing from scratch!

[rajindurbal · Accepted solution]

So I worked with Splunk Support and what I had to do for this error was to:

• Stop Splunk
• rename the current mongo folder to old
• Start Splunk
• And you will see a new mongo folder created with all the components.

[Spinner79]

---

@rajindurbal wrote:

So I worked with Splunk Support and what I had to do for this error was to:

• Stop Splunk
• rename the current mongo folder to old
• Start Splunk
• And you will see a new mongo folder created with all the components.

---

Hi,

I have done renaming the Mongo and recreate new, but still no luck.

Thank you

[vinod743374]

I am Also facing the same issue,

if we do the Procedure you said in the solution will the KV store data is cleared or it will be the same.

can you please confirm once.

[baonguyen78]

I tried this on a distributed splunk setup, and upon restart, mongo.old got removed, kvstore error persists, and mongod isn't running.

[scannon4]

Where is the mongo folder located?

[orezaie]

$Splunk_DB/kvstore/mongo

$SPLUNK_DB , by default, is located in $SPLUNK_HOME/var/lib/splunk

[CarsonZa]

@scannon4 $SPLUNK_HOME\var\lib\splunk\kvstore

[daymauler]

It worked!!! Thanks

[Spinner79]

did you re- generate the mongo?

[woodcock]

Be advised that this approach means that you will be reinitializing from scratch and you will lose ALL KVStore data (you do have a copy of it in old) unless you are in a cluster and you are only doing this on one Search Head!

[ohignett]

Does this mean reconfiguration of apps would be imminent?

[CarsonZa]

@ohignett if they use the kvstore yes, for example Stream. If you were to clean the kvstore you would lose all configurations for that app. In my experience very little apps use the kvstore in this manner.

[vadivel_parames]

Great Stuff!

[nick405060]

wowwwwwwwwwwwwwwwwwwwwwwwwwww

[jdthiele]

This is what I needed to do after rsyncing the entire /opt/splunk folder over to a new file system to move splunk off of the root file system. Thanks for the help!!