topic Why am I experiencing KVStore Failure using Red Hat Linux 7.5 and Splunk 7.3.4? in Knowledge Management

Why am I experiencing KVStore Failure using Red Hat Linux 7.5 and Splunk 7.3.4?

Funderburg78 — Mon, 14 Mar 2022 19:17:13 GMT

W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release. F NETWORK [main] The provided SSL certificate is expired or not yet valid. F - [main] Fatal Assertion 28652 at src/mongo/util/net/ssl_manager.cpp 1145F F F - [main] ***aborting after fassert() failure

I am on a closed network so I copied these errors from other posts and removed their older time stamps. Yes, I have tried removing server.pem and restarting splunk it does nto auto generate a new Server .pem. Yes I followed the attached instructions: https://splunkonbigdata.com/2019/07/03/failed-to-start-kv-store-process-see-mongod-log-and-splunkd-log-for-details/

I do have in server.conf

[sslConfig] caCertFile= $SPLUNK_HOME/etc/auth/cacert.pem caPath=$SPLUNK_HOME/etc/auth enableSplunkdSSL = true serverCert = /opt/splunk/etc/auth/mycerts/myCert.pem SSLRootCAPath = /opt/splunk/etc/auth/mycerts/CA-Chain-Cert.pem

I do not have any Certs listed under [KVStore] section

Not sure if it defaults to use server.pem if not listed or if it defaults to the SSLConfig. The certs in my SSLConfig ARE expired and I cannot get server team to generate new ones. I have a distributed environment. I can create local certs using ./splunk createssl if that helps and move off of the Current CA the enterprise uses since it is needing upgraded anyway.

I am using Red Hat Linux 7.5 and Splunk 7.3.4 and I have Enterprise Security and UBA as well. I first noticed this error after a reboot on the ES Server search server. I then later did a rolling restart on my index cluster and they all give kvstore errors now as well. I do not have experience with Splunk ES or UBA and just arrived at this job a few months ago. They have gone through a tone of quasi splunk admins who had little or no experience with SPLUNK due to difficulty finding splunk admins.

I feel like the servercert in sslconfig of server.conf may be my issue. any help is HIGHLY appreciated!

Yes, I will upvote troubleshooting assistance and answers 😛

Re: KVStore Failure

Funderburg78 — Tue, 06 Apr 2021 11:25:21 GMT

FYI This has been resolved.

Turns out if you utilize the

[sslConfig]

serverca =

servercerts =

Then the KVStore no longer uses server.pem and instead uses the certs assigned in the sslConfig. This was not mentioned anywhere in Documentation. If this post helps you later please extend some Karma 😛

Re: KVStore Failure

lowcrawl — Mon, 14 Mar 2022 14:51:18 GMT

LOL, taking over after you have gone... the other systems certs expired. This post got me to that point... thanks el Hefe!

Re: KVStore Failure

Funderburg78 — Tue, 15 Mar 2022 14:17:51 GMT

Glad to hear it bro!

Re: Why am I experiencing KVStore Failure using Red Hat Linux 7.5 and Splunk 7.3.4?

pavankumarh — Fri, 07 Jul 2023 16:08:20 GMT

https://community.splunk.com/t5/Knowledge-Management/Why-is-KV-Store-certificate-renewal-not-working/td-p/471252

On Windows, you may get the following error message in mongod.log:

Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609

To fix the error that causes mongod to terminate, you need the following in addition to deleting server.pem:

Open Windows certificate management MMC for the local computer ( certlm.msc )
Navigate to Personal > Certificates
Delete any entries named SplunkServerDefaultCert
Restart splunk.