topic Re: KV Store changed status to failed. Failed to start KV Store process. See mongod.log in Splunk Enterprise

KV Store changed status to failed. Failed to start KV Store process. See mongod.log

mohsplunking — Tue, 21 Jan 2025 17:33:11 GMT

Hello Splunker,

After I upgraded to version 9.4 , KV store does not start , I generated a new certificate by renaming server.pem and restarting the splunk , And now I see the following error on mongod.log

[conn937] SSL peer certificate validation failed: self signed certificate in certificate chain
NETWORK [conn937] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:38268 (connection id: 937)

Does anyone have any idea what could be missing ?

Appreciate your inputs in this regard,

Thank you,

Moh

Re: KV Store changed status to failed. Failed to start KV Store process. See mongod.log

mohsplunking — Tue, 21 Jan 2025 17:50:20 GMT

And Splunkd logs has the following error MONGO GB

WARN MongoClient [999733 KVStoreUpgradeStartupThread] - Disabling TLS hostname validation for localhost
ERROR KVStorageProvider [999733 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '127.0.0.1:8191']

Re: KV Store changed status to failed. Failed to start KV Store process. See mongod.log

VatsalJagani — Wed, 22 Jan 2025 19:30:53 GMT

@mohsplunking - Errors definitely seems to be related to SSL certificate file or SSL certificate configuration in Splunk.

* Its more broader topic to tell exactly what's wrong.

* But need to check SSL certs configured on Splunk and then for those SSL files check expiration date and validation of cert file.

* Make sure Splunk config not having any issues.

I hope this helps!!!

Re: KV Store changed status to failed. Failed to start KV Store process. See mongod.log

mserieys_splunk — Wed, 29 Jan 2025 11:46:06 GMT

Hi,
Please can you confirm if you followed the Splunk 9.4 upgrade pre-steps that are documented here?
https://docs.splunk.com/Documentation/Splunk/9.4.0/Installation/AboutupgradingREADTHISFIRST
There is a section on upgrading the kv-store before running the Splunk 9.4 upgrade.
HTH

Re: KV Store changed status to failed. Failed to start KV Store process. See mongod.log

n8o — Wed, 26 Feb 2025 20:09:56 GMT

If you're still experiencing issues, please take a look here https://splunk.my.site.com/customer/s/article/KV-store-status-failed-after-upgrade-to-9-4

The suggestion of concatenating CA certs resolved the errors and Splunk was able to upgrade/initialize kvstore after a restart of splunkd.

Re: KV Store changed status to failed. Failed to start KV Store process. See mongod.log

myitlab42000 — Wed, 26 Feb 2025 23:47:32 GMT

hi,

i don't know if it is the same issue but could you check this requirements. For example, is your cpu supported avx / avx2 instructions, if yes, is it enabled ?

https://docs.splunk.com/Documentation/Splunk/9.4.0/Admin/MigrateKVstore

https://www.mongodb.com/docs/manual/administration/production-notes/

i hope this help