Splunk Enterprise

Why does KV store fail to initialize?


The problem:

My search head is populating with an audit lookup error after upgrading from 9.0.0 to 9.0.2. 

What I've found:

Looking into windows cert mmc on my Splunk server I saw two certs. The self-signed root CA from Splunk, and a cert named SplunkServerDefaultCert below it that is expired. I'm assuming this expired cert is causing the issue and not the actual upgrade itself.

Next, I checked my KVStore status, it's reading "failed." 

Then I checked web.conf, enableSplunkWebSSL = true, there's a password populated in sslPassword, then I ensured privateKeyPath/serverCert/sslRootCAPath had the files in each location as well as checked the expiration dates for each one. The PEM for serverCert is indeed expired. 

What I've done so far:

I renamed the server.pem file to server.pem.back, restarted Splunk and hoped a new cert generated. Didn't work. All that did was prevent the web interface from working. 

Then I went into openssl.conf and inserted "extendedKeyUsage = serverAuth, clientAuth" in the [v3_req] settings and uncommented "req_extensions = v3_req"  in [req]. 

I moved on to openssl to generate a new server cert. Created and signed the new server CSR, verified it, and replaced the  old  server cert w/ the new server PEM. Still didn't work. 

Found $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key, renamed it, restarted splunk, found that a new key was generated, and my KVstore status still reads as "failed." 

Going forward:

Not sure what else I can do to fix this. Given I backed up everything, I restored it all back to square one w/all the OG certs and keys except the openssl.cnf, I left the changes I made stated earlier. 

This is my first time working w/certs, I'm not too savvy w/ any of it, but a lot of the things I did above have all come from other asked questions on this community. 

I think one place I may have made a mistake was signing the server.csr I created. I signed it with the new private.key that was created along with it, not the key that is currently annotated in web.conf. I don't know if that makes a difference, but I can't think of any other reason why the new server.pem  didn't work. 

For reference:

Jeremy describes my exact issue in the below post; however, I do not have the password to the OG splunk cert in the mmc, so I cannot recreate it as he did. 

Windows upgrade from 8.1.1 to 9.0: Why does it fai... - Splunk Community

Additionally, the above case, is the exact issue I am having down to the error codes.

Labels (1)
Tags (3)
0 Karma
1 Solution
Get Updates on the Splunk Community!

Splunk Lantern | Getting Started with Edge Processor, Machine Learning Toolkit ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...