KV Store changed status to failed. Failed to start KV Store process. See mongod.log

[mohsplunking]

Hello Splunker,

After I upgraded to version 9.4 , KV store does not start , I generated a new certificate by renaming server.pem and restarting the splunk , And now I see the following error on mongod.log

[conn937] SSL peer certificate validation failed: self signed certificate in certificate chain
NETWORK [conn937] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:38268 (connection id: 937)

Does anyone have any idea what could be missing ?

Appreciate your inputs in this regard,

Thank you,

Moh

[myitlab42000]

hi,

i don't know if it is the same issue but could you check this requirements. For example, is your cpu supported avx / avx2 instructions, if yes, is it enabled ?

https://docs.splunk.com/Documentation/Splunk/9.4.0/Admin/MigrateKVstore

https://www.mongodb.com/docs/manual/administration/production-notes/

i hope this help

[n8o]

If you're still experiencing issues, please take a look here https://splunk.my.site.com/customer/s/article/KV-store-status-failed-after-upgrade-to-9-4

The suggestion of concatenating CA certs resolved the errors and Splunk was able to upgrade/initialize kvstore after a restart of splunkd.

[mserieys_splunk]

Hi,
Please can you confirm if you followed the Splunk 9.4 upgrade pre-steps that are documented here?
https://docs.splunk.com/Documentation/Splunk/9.4.0/Installation/AboutupgradingREADTHISFIRST
There is a section on upgrading the kv-store before running the Splunk 9.4 upgrade.
HTH

[VatsalJagani]

@mohsplunking  - Errors definitely seems to be related to SSL certificate file or SSL certificate configuration in Splunk.

* Its more broader topic to tell exactly what's wrong.

* But need to check SSL certs configured on Splunk and then for those SSL files check expiration date and validation of cert file.

* Make sure Splunk config not having any issues.

I hope this helps!!!

[mohsplunking]

And Splunkd logs has the following error MONGO GB

WARN MongoClient [999733 KVStoreUpgradeStartupThread] - Disabling TLS hostname validation for localhost
ERROR KVStorageProvider [999733 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '127.0.0.1:8191']