Getting Data In

When upgrading to Splunk enterprise 6.5.0 why am I receiving the error" KV Store changed status to failed, KVStore process terminated"?

sbenamro
New Member

I saw that resolution to the same issue had been posted:
https://answers.splunk.com/answers/457893/after-upgrading-to-650-kv-store-will-not-start.html?utm_so...

yet - my Splunk is running on Windows and not on Linux - this solution is for Linux only.
how do I solve it over Windows ?

0 Karma

splunk68
Path Finder

The solution posted here link text is the right one, even you're running on Windows.

Let's assume your $SPLUNK_HOME folder is C:\Program Files\Splunk\ for example.

Go to C:\Program Files\Splunk\etc\auth and check if the date of your certificate (server.pem) is still valid:

openssl x509 -enddate -noout -in "C:\Program Files\Splunk\etc\auth\server.pem"

If expired, go to C:\Program Files\Splunk\bin and run:

splunk createssl server-cert -d "C:\Program Files\Splunk\etc\auth\server.pem" -n server -c YourDomain -l 2048

That will overwrite your "server.pem" and make it valid 1 more year.
Restart Splunk and you should be fine.

ow111liv
Engager

I had this error on Splunk Enterprise on Linux, I modified the commands for Linux, restarted my server and all is well...

  1. openssl x509 -enddate -noout -in server.pem
    notAfter=May 29 17:13:31 2017 GMT

  2. /opt/splunk/bin/splunk createssl server-cert -d "/opt/splunk/etc/auth" -n server -c YOUR.DOMAIN -l 2048

  • Create certificate server.pem signed by the root CA.
  • Store the server.pem key file locally with your client/server application.
  • Enter a secret passphrase when requested.
  • The passphrase is used to access server.pem in your application.
  • Enter the application's hostname as the Common Name when requested.
  • Enter the root CA passphrase (Getting CA Private Key) to sign the keyfile.
  • The keyfile will expire after one year or sooner if the root CA expires.

Generating a 2048 bit RSA private key
.+++
...................................................+++

writing new private key to 'serverkey.pem'

Signature ok
subject=/CN=YOUR.DOMAINt/O=SplunkUser
Getting CA Private Key
subject= /CN=YOUR.DOMAIN/O=SplunkUser
issuer= /C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
notBefore=Mar 12 15:56:47 2018 GMT
notAfter=Mar 11 15:56:47 2021 GMT

  1. service splunk restart
0 Karma

suryaaruna
New Member

I am using vmware environment setup for testing purpose. i am having the same issue. did follow the procedure towards fixing the same. but, still having same issue. can you please let me know in case the procedure is different for VM setup?

0 Karma

bsachitano
Explorer

The openssl x509 command does not work natively in windows. Is there another way to check via powershell?

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

use splunk cmd openssl args...

maxcburger
New Member

For me, it is due to the cliff function that BDB goes off of when the data set goes out of RAM -- I've been placing a higher value on consistent latency than raw throughput. The BDB issues hark back to the original Dynamo paper where the Amazon folks had a bunch of trouble tuning BDB-JE for consistent performance. When I measured the performance of the Voldemort BDB storage engine a year or 2 ago, I found the same issue here.

0 Karma
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...