Knowledge Management
×
Are you a member of the Splunk Community?
Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
- Find Answers
- :
- Splunk Administration
- :
- Admin Other
- :
- Knowledge Management
- :
- Re: KV Store Process Terminated
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
rajindurbal
Path Finder
07-20-2018
01:21 PM
I have gotten 3 error on the search head. The errors are:
- Failed to start KV Store process. See mongod.log and splunkd.log for details.
- KV Store changed status to failed. KVStore process terminated.
- KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.
The problem only occurs on the search head, but the indexers are fine. This is a windows system. When I restart the search I get an error for "Cannot access appserver directly with appServerPorts configured." After a few minutes, splunk starts "normally". Not sure if the two issues are related. Could really use some help.
1 Solution
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
rajindurbal
Path Finder
01-25-2019
07:59 AM
So I worked with Splunk Support and what I had to do for this error was to:
- Stop Splunk
- rename the current mongo folder to old
- Start Splunk
- And you will see a new mongo folder created with all the components.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
jdthiele
Engager
05-09-2019
11:47 AM
This is what I needed to do after rsyncing the entire /opt/splunk folder over to a new file system to move splunk off of the root file system. Thanks for the help!!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
GregorGoetz
New Member
06-22-2022
07:27 AM
Had the same issue moving /opt from root fs to mounted /opt with larger partition after moving ./splunk folder to that new /opt.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
rajindurbal
Path Finder
08-20-2018
04:06 PM
So it turns out that the problem was for some reason, the mongo and kvstore folders did not have the right permissions. Therefore, splunk could not access them. After changing the permission and rebooting, the problems were resolved.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
mhouse
New Member
12-03-2019
07:15 AM
What were the permissions before you made the change and what permissions settings did you change it to in order to get it to work?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
woodcock
Esteemed Legend
11-22-2019
09:46 AM
What are the correct permissions, exactly?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
CarsonZa
Contributor
07-20-2018
06:07 PM
stop splunkd
delete this file
$SPLUNK_HOME\var\lib\splunk\kvstore\mongo\mongod.lock"
start splunkd
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
rajindurbal
Path Finder
07-20-2018
06:22 PM
Unfortunately, I have already tried this method. The issue continues to persist. After restarting, it starts up again with a new mongod.lock. I appreciate the feedback though.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
CarsonZa
Contributor
07-20-2018
06:32 PM
the lock file will always get remade. So the fact that it comes back is normal. But if you are still experiencing issues with the kvstore staying up then try to clean it using splunk clean kvstore --local
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
CarsonZa
Contributor
07-20-2018
02:05 PM
have you looked at your mongod.log for any errors?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
rajindurbal
Path Finder
07-20-2018
02:58 PM
The mongod.log says "Detected unclean shutdown - \mongod.lock is not empty. ".
Get Updates on the Splunk Community!
Dashboards: Hiding charts while search is being executed and other uses for tokens
There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...
Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...
This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...
Brains, Bytes, and Boston: Learn from the Best at .conf25
When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...
