Splunk Enterprise

Why did Kvstore fail on SHs and indexers?

ssuluguri
Path Finder

Hi team,

KV strore is getting failed frequently on both SHs and indexers after upgrading to 9.0.2 . Can some help if any known resolution .

ssuluguri_0-1680798876042.png

 

Labels (1)
0 Karma
1 Solution

ssuluguri
Path Finder

My issues resolved after performing below steps.

Issue on windows machine .

1.Removed splunk certificates from machine 

2.Removed server.pem 

 

Restarted splunk and KV store READY.

 

Thanks for your time.

View solution in original post

ssuluguri
Path Finder

My issues resolved after performing below steps.

Issue on windows machine .

1.Removed splunk certificates from machine 

2.Removed server.pem 

 

Restarted splunk and KV store READY.

 

Thanks for your time.

woodcock
Esteemed Legend

There are innumerable bugs causing uncontrolled KVStore grown in all versions of Splunk.  It is truly maddening.  What I sometimes do in cloud environments is create a KVSTORE directory in /run stop splunk, move the files from wherever they are growing uncontrollably to /run/KVSTORE, then create delete the original directory, then replace the source directory with a soft link pointing to /run/KVSTORE.  Yes, this means that when the server reboots, all of these files are lost, but so far as I can tell, theses are not strictly necessary and will be recreated in that situation.  Splunk really needs to re-engineer KVSTORE.

richgalloway
SplunkTrust
SplunkTrust

KVStore should be disabled on all indexers because it is not used.

What release did you upgrade from?  Have you upgraded the KVStore engine to wiredTiger and the version to 4.2 per https://docs.splunk.com/Documentation/Splunk/9.0.4/Admin/MigrateKVstore?

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...