I recently realized that we've been getting the following error messages for months, and have never been able to fix them:
"Failed to start KV Store process. See mongod.log and splunkd.log for details."
I've found other questions that answer how to fix this but my question is: what does KV Store actually do? We've been running our Spunk without it for several months, so what happens if we don't fix this/what part of Splunk hasn't been working correctly under the hood this whole time?
Here are the docs: http://docs.splunk.com/Documentation/Splunk/latest/Admin/AboutKVstore and http://dev.splunk.com/view/SP-CAAAEY7. Those docs go through some of the uses of the KVstore. It is used by a variety of apps and can be used as an alternative to CSV lookups as well. There are tradeoffs between using CSV and KVstore lookups that can be considered. You may not be using an app using the KVstore which may be why you have not noticed any negative effects. It can be disabled in server.conf altogether, but would be something to consider and may be something you want running for future lookups and app configurations.
I would give all my karma to anyone that could explain this to me... I keep getting pointed to docs that read like Latin to me, making no sense in practical terms. I'm a security/intell analysts, not a data-scientist; been using Splunk for over 8 years with a couple of certs under my belt, and have seen tons of KV store errors in my logs since they introduced them -- and mongod.log doesn't ever mention it (or help)... My Karma-Kingdom for a (good, pragmatic) explanation!
Here are the docs: http://docs.splunk.com/Documentation/Splunk/latest/Admin/AboutKVstore and http://dev.splunk.com/view/SP-CAAAEY7. Those docs go through some of the uses of the KVstore. It is used by a variety of apps and can be used as an alternative to CSV lookups as well. There are tradeoffs between using CSV and KVstore lookups that can be considered. You may not be using an app using the KVstore which may be why you have not noticed any negative effects. It can be disabled in server.conf altogether, but would be something to consider and may be something you want running for future lookups and app configurations.