topic Re: KV store failing to start in 9.4.3 in Splunk Enterprise

KV store failing to start in 9.4.3

Namo — Fri, 20 Jun 2025 08:26:28 GMT

Hello Team,

We are on Linux and Post upgrade to splunk 9.4.3, KV store is failing.I have followed few recommendations given in the community for the related issue,but they are not working .Below is the mongod.log

SSL peer certificate validation failed: self signed certificate in certificate chain
2025-06-20T08:09:03.925Z I NETWORK [conn638] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:54188 (connection id: 638)

This error can be bypassed if we add the below stanza in server.conf, though it is a workaround only.
enableSplunkdSSL = false

Any other inputs is appreciated.

Re: KV store failing to start in 9.4.3

tej57 — Fri, 20 Jun 2025 08:54:23 GMT

Hello @Namo,

There can be multiple reasons for KVStore failure. What version of Splunk did you upgrade from? Also, did you check the expiry of the certificate used by kvstore? Setting enabledSplunkdSSL to false will disconnect secure communication internally throughout the Splunk deployment wherever management port is being used.

Thanks,
Tejas.

Re: KV store failing to start in 9.4.3

Namo — Fri, 20 Jun 2025 09:18:37 GMT

Hello @tej57
It was upgraded from 9.2 to 9.4. The cert is not expired ,is valid for another few days .
the server cert is a combined cert

Re: KV store failing to start in 9.4.3

kiran_panchavat — Fri, 20 Jun 2025 11:01:11 GMT

@Namo

Could you please confirm the upgrade path — specifically, from which version to which version Splunk was upgraded?

Please note that you must first upgrade to the KV Store server version 4.2.x before proceeding with an upgrade to Splunk Enterprise 9.4.x or higher. For detailed instructions on updating to KV Store version 4.2.x (applicable to Splunk Enterprise versions 9.0.x through 9.3.x), refer to the official documentation: Migrate the KV store storage engine in the https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.3/administer-the-app-key-value-store/migrate-the-kv-store-storage-engine

We strongly recommend reviewing this guide to ensure a successful upgrade path and avoid issues like the one you're encountering.

https://docs.splunk.com/Documentation/Splunk/9.3.2/Admin/MigrateKVstore

Re: KV store failing to start in 9.4.3

kiran_panchavat — Fri, 20 Jun 2025 11:03:12 GMT

@Namo

Please can you confirm if you followed the Splunk 9.4 upgrade pre-steps that are documented here?
https://docs.splunk.com/Documentation/Splunk/9.4.0/Installation/AboutupgradingREADTHISFIRST
There is a section on upgrading the kv-store before running the Splunk 9.4 upgrade.

Reference:

https://splunk.my.site.com/customer/s/article/KV-store-status-failed-after-upgrade-to-9-4?

Re: KV store failing to start in 9.4.3

Namo — Fri, 20 Jun 2025 12:05:47 GMT

The KV sore was already on 4.2 before the upgrade.Hence went ahead with the splunk upgrade.
The readme file was reviewed before the upgrade.
The error states
Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:58948

Does it mean we need to review the certificate steps also.
Currently the server cert is issued by Digicert and cacert doesnt have Digicert as list of trusted list

Re: KV store failing to start in 9.4.3

Namo — Fri, 20 Jun 2025 12:10:05 GMT

Hi @kiran_panchavat

There is no folder with the below name in the $SPLUNK_HOME/etc/apps folder

100_<stackname>_splunkcloud

Re: KV store failing to start in 9.4.3

livehybrid — Fri, 20 Jun 2025 12:28:14 GMT

Hi @Namo

Have you got custom SSL in use on your Splunk instance?

One thing you could check is running:

$SPLUNK_HOME/bin/splunk cmd btool server list kvstore

Check specifically for sslVerifyServerCert - Is this true? If so try setting to false and restart Splunk to see if this resolves the issue temporarily - if it does then at least you can get the service back up and then focus on how you can get the SSL certs in working state without having to set sslVerifyServerCert to false.

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: KV store failing to start in 9.4.3

LAME-Creations — Fri, 20 Jun 2025 13:40:56 GMT

We recently had the exact same issue on our environment. It was all related to expiring certs. It sounds like you have already looked into this, but I just want to echo that when the cert expires or can't be found, you can get this very behavior and it took a lot longer than I want to admit for our team to figure out that this was the reason all of our KV stores were failing.

Re: KV store failing to start in 9.4.3

Namo — Fri, 20 Jun 2025 14:10:35 GMT

@livehybrid
I will check,Thanks

Re: KV store failing to start in 9.4.3

tscroggins — Sat, 21 Jun 2025 15:16:53 GMT

Hi @Namo,

Make sure $SPLUNK_HOME/etc/auth/cacert.pem contains all certificates in the trust chain. If you're using a self-signed certificate, add this certificate to cacert.pem. If you've changed the name or location of the file, update the new file.

If you're also attempting a KV store upgrade, check the prerequisites at https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/administer-the-app-key-value-store/upgrade-the-kv-store-server-version#ariaid-title2 as others have recommended. Also note that your private key must be encrypted with the correct sslPassword value in server.conf for a KV store upgrade to succeed. When using a blank/empty password, you'll see a message similar to the following in splunkd.log:

06-21-2025 00:00:00.000 -0000 WARN KVStoreUpgradeToolTLS [133719 KVStoreConfigurationThread] - Incomplete TLS settings detected, skipping creation of KVStore TLS credentials file!

Re: KV store failing to start in 9.4.3

Namo — Mon, 23 Jun 2025 12:34:33 GMT

Hi @tscroggins
I have appended intermediate and root cert to the cacert.pem .After this error is not observed.

Re: KV store failing to start in 9.4.3

splunker686 — Thu, 27 Nov 2025 01:28:09 GMT

We have encountered similar kvstore issue once upgraded to splunk 9.4.6. Splunk support recommended to use this kvcertverify tool:

https://github.com/splunk/support-scripts/blob/main/kvcertverify/README.md

And it did help in resolving our issue:

Server.conf snippet for reference:

[sslConfig]
serverCert = $SPLUNK_HOME/etc/auth/mycerts/splunk-hf.pem
caCertFile = $SPLUNK_HOME/etc/auth/mycerts/splunk-hf.pem
caPath = $SPLUNK_HOME/etc/auth/mycerts/splunk-hf.pem

[kvstore]
disabled = false
serverCert = $SPLUNK_HOME/etc/auth/mycerts/splunk-hf.pem
caCertFile = $SPLUNK_HOME/etc/auth/mycerts/splunk-hf.pem

Here is the structure of our certificate chain (.pem) file:

$ grep "BEGIN\|END" splunk-hf.pem
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

Note: The .pem file was created as per this splunk doc