I saw that resolution to the same issue had been posted:
https://answers.splunk.com/answers/457893/after-upgrading-to-650-kv-store-will-not-start.html?utm_so...
yet - my Splunk is running on Windows and not on Linux - this solution is for Linux only.
how do I solve it over Windows ?
The solution posted here link text is the right one, even you're running on Windows.
Let's assume your $SPLUNK_HOME folder is C:\Program Files\Splunk\ for example.
Go to C:\Program Files\Splunk\etc\auth and check if the date of your certificate (server.pem) is still valid:
openssl x509 -enddate -noout -in "C:\Program Files\Splunk\etc\auth\server.pem"
If expired, go to C:\Program Files\Splunk\bin and run:
splunk createssl server-cert -d "C:\Program Files\Splunk\etc\auth\server.pem" -n server -c YourDomain -l 2048
That will overwrite your "server.pem" and make it valid 1 more year.
Restart Splunk and you should be fine.
I had this error on Splunk Enterprise on Linux, I modified the commands for Linux, restarted my server and all is well...
openssl x509 -enddate -noout -in server.pem
notAfter=May 29 17:13:31 2017 GMT
/opt/splunk/bin/splunk createssl server-cert -d "/opt/splunk/etc/auth" -n server -c YOUR.DOMAIN -l 2048
Generating a 2048 bit RSA private key
.+++
...................................................+++
Signature ok
subject=/CN=YOUR.DOMAINt/O=SplunkUser
Getting CA Private Key
subject= /CN=YOUR.DOMAIN/O=SplunkUser
issuer= /C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
notBefore=Mar 12 15:56:47 2018 GMT
notAfter=Mar 11 15:56:47 2021 GMT
I am using vmware environment setup for testing purpose. i am having the same issue. did follow the procedure towards fixing the same. but, still having same issue. can you please let me know in case the procedure is different for VM setup?
The openssl x509 command does not work natively in windows. Is there another way to check via powershell?
use splunk cmd openssl args...
For me, it is due to the cliff function that BDB goes off of when the data set goes out of RAM -- I've been placing a higher value on consistent latency than raw throughput. The BDB issues hark back to the original Dynamo paper where the Amazon folks had a bunch of trouble tuning BDB-JE for consistent performance. When I measured the performance of the Voldemort BDB storage engine a year or 2 ago, I found the same issue here.