I have gotten 3 error on the search head. The errors are:
The problem only occurs on the search head, but the indexers are fine. This is a windows system. When I restart the search I get an error for "Cannot access appserver directly with appServerPorts configured." After a few minutes, splunk starts "normally". Not sure if the two issues are related. Could really use some help.
So I worked with Splunk Support and what I had to do for this error was to:
Had the same issue moving /opt from root fs to mounted /opt with larger partition after moving ./splunk folder to that new /opt.
So it turns out that the problem was for some reason, the mongo and kvstore folders did not have the right permissions. Therefore, splunk could not access them. After changing the permission and rebooting, the problems were resolved.
What were the permissions before you made the change and what permissions settings did you change it to in order to get it to work?
What are the correct permissions, exactly?
stop splunkd
delete this file
$SPLUNK_HOME\var\lib\splunk\kvstore\mongo\mongod.lock"
start splunkd
Unfortunately, I have already tried this method. The issue continues to persist. After restarting, it starts up again with a new mongod.lock. I appreciate the feedback though.
the lock file will always get remade. So the fact that it comes back is normal. But if you are still experiencing issues with the kvstore staying up then try to clean it using splunk clean kvstore --local
have you looked at your mongod.log for any errors?
The mongod.log says "Detected unclean shutdown - \mongod.lock is not empty. ".