Knowledge Management

Why am I experiencing KVStore Failure using Red Hat Linux 7.5 and Splunk 7.3.4?

Funderburg78
Path Finder

 

 

W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
F NETWORK [main] The provided SSL certificate is expired or not yet valid.
F - [main] Fatal Assertion 28652 at src/mongo/util/net/ssl_manager.cpp 1145F F F - [main] 
***aborting after fassert() failure

 

 

 I am on a closed network so I copied these errors from other posts and removed their older time stamps.   Yes, I have tried removing server.pem and restarting splunk it does nto auto generate a new Server .pem.  Yes I followed the attached instructions: https://splunkonbigdata.com/2019/07/03/failed-to-start-kv-store-process-see-mongod-log-and-splunkd-l... 

I do have in server.conf

[sslConfig]

caCertFile= $SPLUNK_HOME/etc/auth/cacert.pem

caPath=$SPLUNK_HOME/etc/auth

enableSplunkdSSL = true

serverCert = /opt/splunk/etc/auth/mycerts/myCert.pem

SSLRootCAPath = /opt/splunk/etc/auth/mycerts/CA-Chain-Cert.pem

I do not have any Certs listed under [KVStore] section 

Not sure if it defaults to use server.pem if not listed or if it defaults to the SSLConfig.  The certs in my SSLConfig ARE expired and I cannot get server team to generate new ones.  I have a distributed environment.  I can create local certs using ./splunk createssl if that helps and move off of the Current CA the enterprise uses since it is needing upgraded anyway.  

I am using Red Hat Linux 7.5 and Splunk 7.3.4 and I have Enterprise Security and UBA as well.  I first noticed this error after a reboot on the ES Server search server.  I then later did a rolling restart on my index cluster and they all give kvstore errors now as well.  I do not have experience with Splunk ES or UBA and just arrived at this job a few months ago.  They have gone through a tone of quasi splunk admins who had little or no experience with SPLUNK due to difficulty finding splunk admins.  

I feel like the servercert in sslconfig of server.conf may be my issue.  any help is HIGHLY appreciated!

Yes, I will upvote troubleshooting assistance and answers 😛

Labels (1)
0 Karma
1 Solution

Funderburg78
Path Finder

FYI This has been resolved.

 

Turns out if you utilize the 

[sslConfig]

serverca =

servercerts =

Then the KVStore no longer uses server.pem and instead uses the certs assigned in the sslConfig.  This was not mentioned anywhere in Documentation.  If this post helps you later please extend some Karma 😛

View solution in original post

pavankumarh
Path Finder

https://community.splunk.com/t5/Knowledge-Management/Why-is-KV-Store-certificate-renewal-not-working... 

On Windows, you may get the following error message in mongod.log:

Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609

To fix the error that causes mongod to terminate, you need the following in addition to deleting server.pem:

  1. Open Windows certificate management MMC for the local computer ( certlm.msc )
  2. Navigate to Personal > Certificates
  3. Delete any entries named SplunkServerDefaultCert
  4. Restart splunk. 
0 Karma

lowcrawl
Explorer

LOL, taking over after you have gone... the other systems certs expired.  This post got me to that point... thanks el Hefe!

0 Karma

Funderburg78
Path Finder

Glad to hear it bro!

0 Karma

Funderburg78
Path Finder

FYI This has been resolved.

 

Turns out if you utilize the 

[sslConfig]

serverca =

servercerts =

Then the KVStore no longer uses server.pem and instead uses the certs assigned in the sslConfig.  This was not mentioned anywhere in Documentation.  If this post helps you later please extend some Karma 😛

Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...