Why did Kvstore fail on SHs and indexers?

[ssuluguri]

Hi team,

KV strore is getting failed frequently on both SHs and indexers after upgrading to 9.0.2 . Can some help if any known resolution .

[ssuluguri · Accepted solution]

My issues resolved after performing below steps.

Issue on windows machine .

1.Removed splunk certificates from machine 

2.Removed server.pem 

Restarted splunk and KV store READY.

Thanks for your time.

[woodcock]

There are innumerable bugs causing uncontrolled KVStore grown in all versions of Splunk.  It is truly maddening.  What I sometimes do in cloud environments is create a KVSTORE directory in /run stop splunk, move the files from wherever they are growing uncontrollably to /run/KVSTORE, then create delete the original directory, then replace the source directory with a soft link pointing to /run/KVSTORE.  Yes, this means that when the server reboots, all of these files are lost, but so far as I can tell, theses are not strictly necessary and will be recreated in that situation.  Splunk really needs to re-engineer KVSTORE.

[richgalloway]

KVStore should be disabled on all indexers because it is not used.

What release did you upgrade from?  Have you upgraded the KVStore engine to wiredTiger and the version to 4.2 per https://docs.splunk.com/Documentation/Splunk/9.0.4/Admin/MigrateKVstore?