Splunk Enterprise

KV Store status failed after upgrade to 8.0.4.1?

krylov
Explorer

Good afternoon!

 After upgrade to version 8.0.4.1 from 8.0.1 KV Store status failed.

 

Failed to start KV Store process. See mongod.log and splunkd.log for details.
15.06.2020, 21:12:09
KV Store changed status to failed. KVStore process terminated..
15.06.2020, 21:12:08
KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.

 

./splunk show kvstore-status

This member:

                                 backupRestoreStatus : Ready

                                      disabled : 0

                                          guid : A5CBF13E-6100-40DA-8C51-541893E5A0A0

                                          port : 8191

                                    standalone : 1

                                        status : failed

 

/data/splunk/var/log/splunk/mongod.log

2020-06-16T05:25:17.315Z W CONTROL  [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.

 2020-06-16T05:25:17.323Z F NETWORK  [main] The provided SSL certificate is expired or not yet valid.

 2020-06-16T05:25:17.323Z F -        [main] Fatal Assertion 28652 at src/mongo/util/net/ssl_manager.cpp 1214

 2020-06-16T05:25:17.323Z F -        [main]

 ***aborting after fassert() failure

 

 

/data/splunk/var/log/splunk/splunkd.log

ERROR KVStoreAdminHandler - An error occurred.

ERROR KVStorageProvider - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling ismaster on '127.0.0.1:8191']

ERROR KVStoreIntrospection - failed to get introspection data

 

 

Could you help me figure out how to fix this?

Thanks in advance!

 

 

Labels (1)
1 Solution

krylov
Explorer

The question isn't longer relevant. The problem solved by the procedure described in this ask

https://community.splunk.com/t5/Knowledge-Management/After-upgrading-to-6-5-0-KV-Store-will-not-star...

View solution in original post

0 Karma

pavankumarh
Path Finder

https://community.splunk.com/t5/Knowledge-Management/Why-is-KV-Store-certificate-renewal-not-working... 

On Windows, you may get the following error message in mongod.log:

Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609

To fix the error that causes mongod to terminate, you need the following in addition to deleting server.pem:

Open Windows certificate management MMC for the local computer ( certlm.msc )
Navigate to Personal > Certificates
Delete any entries named SplunkServerDefaultCert

Restart splunk. 

0 Karma

spodda01da
Path Finder

@krylov : Did you manage to find the solution... I have similar issue where server.pem certificate is valid and also regenerated mongo DB.

0 Karma

krylov
Explorer

The question isn't longer relevant. The problem solved by the procedure described in this ask

https://community.splunk.com/t5/Knowledge-Management/After-upgrading-to-6-5-0-KV-Store-will-not-star...

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...