Why am I experiencing KVStore Failure using Red Hat Linux 7.5 and Splunk 7.3.4?

[Funderburg78]

W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
F NETWORK [main] The provided SSL certificate is expired or not yet valid.
F - [main] Fatal Assertion 28652 at src/mongo/util/net/ssl_manager.cpp 1145F F F - [main] 
***aborting after fassert() failure

 I am on a closed network so I copied these errors from other posts and removed their older time stamps.   Yes, I have tried removing server.pem and restarting splunk it does nto auto generate a new Server .pem.  Yes I followed the attached instructions: https://splunkonbigdata.com/2019/07/03/failed-to-start-kv-store-process-see-mongod-log-and-splunkd-l... 

I do have in server.conf

[sslConfig]

caCertFile= $SPLUNK_HOME/etc/auth/cacert.pem

caPath=$SPLUNK_HOME/etc/auth

enableSplunkdSSL = true

serverCert = /opt/splunk/etc/auth/mycerts/myCert.pem

SSLRootCAPath = /opt/splunk/etc/auth/mycerts/CA-Chain-Cert.pem

I do not have any Certs listed under [KVStore] section 

Not sure if it defaults to use server.pem if not listed or if it defaults to the SSLConfig.  The certs in my SSLConfig ARE expired and I cannot get server team to generate new ones.  I have a distributed environment.  I can create local certs using ./splunk createssl if that helps and move off of the Current CA the enterprise uses since it is needing upgraded anyway.  

I am using Red Hat Linux 7.5 and Splunk 7.3.4 and I have Enterprise Security and UBA as well.  I first noticed this error after a reboot on the ES Server search server.  I then later did a rolling restart on my index cluster and they all give kvstore errors now as well.  I do not have experience with Splunk ES or UBA and just arrived at this job a few months ago.  They have gone through a tone of quasi splunk admins who had little or no experience with SPLUNK due to difficulty finding splunk admins.  

I feel like the servercert in sslconfig of server.conf may be my issue.  any help is HIGHLY appreciated!

Yes, I will upvote troubleshooting assistance and answers 😛

[pavankumarh]

https://community.splunk.com/t5/Knowledge-Management/Why-is-KV-Store-certificate-renewal-not-working... 

On Windows, you may get the following error message in mongod.log:

Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609

To fix the error that causes mongod to terminate, you need the following in addition to deleting server.pem:

1. Open Windows certificate management MMC for the local computer ( certlm.msc )
2. Navigate to Personal > Certificates
3. Delete any entries named SplunkServerDefaultCert
4. Restart splunk. 

[lowcrawl]

LOL, taking over after you have gone... the other systems certs expired.  This post got me to that point... thanks el Hefe!

[Funderburg78]

Glad to hear it bro!

[Funderburg78 · Accepted solution]

FYI This has been resolved.

Turns out if you utilize the 

[sslConfig]

serverca =

servercerts =

Then the KVStore no longer uses server.pem and instead uses the certs assigned in the sslConfig.  This was not mentioned anywhere in Documentation.  If this post helps you later please extend some Karma 😛