Hello Team,
We are on Linux and Post upgrade to splunk 9.4.3, KV store is failing.I have followed few recommendations given in the community for the related issue,but they are not working .Below is the mongod.log
SSL peer certificate validation failed: self signed certificate in certificate chain
2025-06-20T08:09:03.925Z I NETWORK [conn638] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:54188 (connection id: 638)
This error can be bypassed if we add the below stanza in server.conf, though it is a workaround only.
enableSplunkdSSL = false
Any other inputs is appreciated.
Hi @Namo,
Make sure $SPLUNK_HOME/etc/auth/cacert.pem contains all certificates in the trust chain. If you're using a self-signed certificate, add this certificate to cacert.pem. If you've changed the name or location of the file, update the new file.
If you're also attempting a KV store upgrade, check the prerequisites at https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/administer-the-app-key-valu... as others have recommended. Also note that your private key must be encrypted with the correct sslPassword value in server.conf for a KV store upgrade to succeed. When using a blank/empty password, you'll see a message similar to the following in splunkd.log:
06-21-2025 00:00:00.000 -0000 WARN KVStoreUpgradeToolTLS [133719 KVStoreConfigurationThread] - Incomplete TLS settings detected, skipping creation of KVStore TLS credentials file!
Hi @tscroggins
I have appended intermediate and root cert to the cacert.pem .After this error is not observed.
Hi @Namo
Have you got custom SSL in use on your Splunk instance?
One thing you could check is running:
$SPLUNK_HOME/bin/splunk cmd btool server list kvstore
Check specifically for sslVerifyServerCert - Is this true? If so try setting to false and restart Splunk to see if this resolves the issue temporarily - if it does then at least you can get the service back up and then focus on how you can get the SSL certs in working state without having to set sslVerifyServerCert to false.
🌟 Did this answer help you? If so, please consider:
Your feedback encourages the volunteers in this community to continue contributing
@livehybrid
I will check,Thanks
Could you please confirm the upgrade path — specifically, from which version to which version Splunk was upgraded?
Please note that you must first upgrade to the KV Store server version 4.2.x before proceeding with an upgrade to Splunk Enterprise 9.4.x or higher. For detailed instructions on updating to KV Store version 4.2.x (applicable to Splunk Enterprise versions 9.0.x through 9.3.x), refer to the official documentation: Migrate the KV store storage engine in the https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.3/administer-the-app-key-valu...
We strongly recommend reviewing this guide to ensure a successful upgrade path and avoid issues like the one you're encountering.
https://docs.splunk.com/Documentation/Splunk/9.3.2/Admin/MigrateKVstore
The KV sore was already on 4.2 before the upgrade.Hence went ahead with the splunk upgrade.
The readme file was reviewed before the upgrade.
The error states
Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:58948
Does it mean we need to review the certificate steps also.
Currently the server cert is issued by Digicert and cacert doesnt have Digicert as list of trusted list
Please can you confirm if you followed the Splunk 9.4 upgrade pre-steps that are documented here?
https://docs.splunk.com/Documentation/Splunk/9.4.0/Installation/AboutupgradingREADTHISFIRST
There is a section on upgrading the kv-store before running the Splunk 9.4 upgrade.
Reference:
https://splunk.my.site.com/customer/s/article/KV-store-status-failed-after-upgrade-to-9-4?
Hi @kiran_panchavat
There is no folder with the below name in the $SPLUNK_HOME/etc/apps folder
100_<stackname>_splunkcloud
Hello @Namo,
There can be multiple reasons for KVStore failure. What version of Splunk did you upgrade from? Also, did you check the expiry of the certificate used by kvstore? Setting enabledSplunkdSSL to false will disconnect secure communication internally throughout the Splunk deployment wherever management port is being used.
Thanks,
Tejas.
We recently had the exact same issue on our environment. It was all related to expiring certs. It sounds like you have already looked into this, but I just want to echo that when the cert expires or can't be found, you can get this very behavior and it took a lot longer than I want to admit for our team to figure out that this was the reason all of our KV stores were failing.
Hello @tej57
It was upgraded from 9.2 to 9.4. The cert is not expired ,is valid for another few days .
the server cert is a combined cert