Getting Data In

When upgrading to Splunk enterprise 6.5.0 why am I receiving the error" KV Store changed status to failed, KVStore process terminated"?

New Member

I saw that resolution to the same issue had been posted:
https://answers.splunk.com/answers/457893/after-upgrading-to-650-kv-store-will-not-start.html?utm_so...

yet - my Splunk is running on Windows and not on Linux - this solution is for Linux only.
how do I solve it over Windows ?

0 Karma

Path Finder

The solution posted here link text is the right one, even you're running on Windows.

Let's assume your $SPLUNK_HOME folder is C:\Program Files\Splunk\ for example.

Go to C:\Program Files\Splunk\etc\auth and check if the date of your certificate (server.pem) is still valid:

openssl x509 -enddate -noout -in "C:\Program Files\Splunk\etc\auth\server.pem"

If expired, go to C:\Program Files\Splunk\bin and run:

splunk createssl server-cert -d "C:\Program Files\Splunk\etc\auth\server.pem" -n server -c YourDomain -l 2048

That will overwrite your "server.pem" and make it valid 1 more year.
Restart Splunk and you should be fine.

Engager

I had this error on Splunk Enterprise on Linux, I modified the commands for Linux, restarted my server and all is well...

  1. openssl x509 -enddate -noout -in server.pem
    notAfter=May 29 17:13:31 2017 GMT

  2. /opt/splunk/bin/splunk createssl server-cert -d "/opt/splunk/etc/auth" -n server -c YOUR.DOMAIN -l 2048

  • Create certificate server.pem signed by the root CA.
  • Store the server.pem key file locally with your client/server application.
  • Enter a secret passphrase when requested.
  • The passphrase is used to access server.pem in your application.
  • Enter the application's hostname as the Common Name when requested.
  • Enter the root CA passphrase (Getting CA Private Key) to sign the keyfile.
  • The keyfile will expire after one year or sooner if the root CA expires.

Generating a 2048 bit RSA private key
.+++
...................................................+++

writing new private key to 'serverkey.pem'

Signature ok
subject=/CN=YOUR.DOMAINt/O=SplunkUser
Getting CA Private Key
subject= /CN=YOUR.DOMAIN/O=SplunkUser
issuer= /C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
notBefore=Mar 12 15:56:47 2018 GMT
notAfter=Mar 11 15:56:47 2021 GMT

  1. service splunk restart
0 Karma

New Member

I am using vmware environment setup for testing purpose. i am having the same issue. did follow the procedure towards fixing the same. but, still having same issue. can you please let me know in case the procedure is different for VM setup?

0 Karma

Explorer

The openssl x509 command does not work natively in windows. Is there another way to check via powershell?

0 Karma

Splunk Employee
Splunk Employee

use splunk cmd openssl args...

New Member

For me, it is due to the cliff function that BDB goes off of when the data set goes out of RAM -- I've been placing a higher value on consistent latency than raw throughput. The BDB issues hark back to the original Dynamo paper where the Amazon folks had a bunch of trouble tuning BDB-JE for consistent performance. When I measured the performance of the Voldemort BDB storage engine a year or 2 ago, I found the same issue here.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!