Why does KV store fail to initialize?

[dwthomas16]

The problem:

My search head is populating with an audit lookup error after upgrading from 9.0.0 to 9.0.2. 

What I've found:

Looking into windows cert mmc on my Splunk server I saw two certs. The self-signed root CA from Splunk, and a cert named SplunkServerDefaultCert below it that is expired. I'm assuming this expired cert is causing the issue and not the actual upgrade itself.

Next, I checked my KVStore status, it's reading "failed." 

Then I checked web.conf, enableSplunkWebSSL = true, there's a password populated in sslPassword, then I ensured privateKeyPath/serverCert/sslRootCAPath had the files in each location as well as checked the expiration dates for each one. The PEM for serverCert is indeed expired. 

What I've done so far:

I renamed the server.pem file to server.pem.back, restarted Splunk and hoped a new cert generated. Didn't work. All that did was prevent the web interface from working. 

Then I went into openssl.conf and inserted "extendedKeyUsage = serverAuth, clientAuth" in the [v3_req] settings and uncommented "req_extensions = v3_req"  in [req]. 

I moved on to openssl to generate a new server cert. Created and signed the new server CSR, verified it, and replaced the  old  server cert w/ the new server PEM. Still didn't work. 

Found $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key, renamed it, restarted splunk, found that a new key was generated, and my KVstore status still reads as "failed." 

Going forward:

Not sure what else I can do to fix this. Given I backed up everything, I restored it all back to square one w/all the OG certs and keys except the openssl.cnf, I left the changes I made stated earlier. 

This is my first time working w/certs, I'm not too savvy w/ any of it, but a lot of the things I did above have all come from other asked questions on this community. 

I think one place I may have made a mistake was signing the server.csr I created. I signed it with the new private.key that was created along with it, not the key that is currently annotated in web.conf. I don't know if that makes a difference, but I can't think of any other reason why the new server.pem  didn't work. 

For reference:

Jeremy describes my exact issue in the below post; however, I do not have the password to the OG splunk cert in the mmc, so I cannot recreate it as he did. 

Windows upgrade from 8.1.1 to 9.0: Why does it fai... - Splunk Community

Additionally, the above case, is the exact issue I am having down to the error codes.

[dwthomas16 · Accepted solution]

Solution here: https://community.splunk.com/t5/Knowledge-Management/Kvstore-Status-failed/m-p/656113#M9664