I had the following alerts after I restarted Splunk from the web interface. These alerts took place on May 5th and I haven't seen them come back.
Failed to start KV Store process. See mongod.log and splunkd.log for details. KV Store changed status to failed. KVStore process terminated.. KV Store process terminated abnormally (exit code 1, status exited with code 1). See mongod.log and splunkd.log for details.
I checked the "/opt/splunk/var/lib/splunk/kvstore/mongo$" the permissions are set to "splunk: splunk"
I see the "March 1 2017 splunk.key" should that be rotated or something? Should I restart Splunk and see if those errors come back?