- Splunk Answers
- :
- Splunk Administration
- :
- Knowledge Management
- :
- Re: KV Store Process Terminated
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have gotten 3 error on the search head. The errors are:
- Failed to start KV Store process. See mongod.log and splunkd.log for details.
- KV Store changed status to failed. KVStore process terminated.
- KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.
The problem only occurs on the search head, but the indexers are fine. This is a windows system. When I restart the search I get an error for "Cannot access appserver directly with appServerPorts configured." After a few minutes, splunk starts "normally". Not sure if the two issues are related. Could really use some help.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So I worked with Splunk Support and what I had to do for this error was to:
- Stop Splunk
- rename the current mongo folder to old
- Start Splunk
- And you will see a new mongo folder created with all the components.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Had the same issue moving /opt from root fs to mounted /opt with larger partition after moving ./splunk folder to that new /opt.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So it turns out that the problem was for some reason, the mongo and kvstore folders did not have the right permissions. Therefore, splunk could not access them. After changing the permission and rebooting, the problems were resolved.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What were the permissions before you made the change and what permissions settings did you change it to in order to get it to work?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What are the correct permissions, exactly?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
stop splunkd
delete this file
$SPLUNK_HOME\var\lib\splunk\kvstore\mongo\mongod.lock"
start splunkd
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Unfortunately, I have already tried this method. The issue continues to persist. After restarting, it starts up again with a new mongod.lock. I appreciate the feedback though.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
the lock file will always get remade. So the fact that it comes back is normal. But if you are still experiencing issues with the kvstore staying up then try to clean it using splunk clean kvstore --local
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
have you looked at your mongod.log for any errors?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The mongod.log says "Detected unclean shutdown - \mongod.lock is not empty. ".
Announcing Scheduled Export GA for Dashboard Studio
Extending Observability Content to Splunk Cloud
More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!
