Splunk Enterprise

KV store failing to start in 9.4.3

Namo
Explorer

Hello Team,

 

We are on Linux and Post upgrade to splunk 9.4.3, KV store is failing.I have followed few recommendations given in the community  for the related issue,but they are not working .Below is the mongod.log

SSL peer certificate validation failed: self signed certificate in certificate chain
2025-06-20T08:09:03.925Z I NETWORK [conn638] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:54188 (connection id: 638)

This error can be bypassed if we add the below stanza  in server.conf, though it is a workaround only.
enableSplunkdSSL = false

Any other inputs is appreciated.

Labels (1)
0 Karma

tscroggins
Influencer

Hi @Namo,

Make sure $SPLUNK_HOME/etc/auth/cacert.pem contains all certificates in the trust chain. If you're using a self-signed certificate, add this certificate to cacert.pem. If you've changed the name or location of the file, update the new file.

If you're also attempting a KV store upgrade, check the prerequisites at https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/administer-the-app-key-valu... as others have recommended. Also note that your private key must be encrypted with the correct sslPassword value in server.conf for a KV store upgrade to succeed. When using a blank/empty password, you'll see a message similar to the following in splunkd.log:

06-21-2025 00:00:00.000 -0000 WARN  KVStoreUpgradeToolTLS [133719 KVStoreConfigurationThread] - Incomplete TLS settings detected, skipping creation of KVStore TLS credentials file!

 

0 Karma

Namo
Explorer

Hi @tscroggins 
I have appended intermediate  and root cert to the cacert.pem .After this error is not observed.

livehybrid
Super Champion

Hi @Namo 

Have you got custom SSL in use on your Splunk instance? 

One thing you could check is running:

$SPLUNK_HOME/bin/splunk cmd btool server list kvstore

Check specifically for sslVerifyServerCert - Is this true? If so try setting to false and restart Splunk to see if this resolves the issue temporarily - if it does then at least you can get the service back up and then focus on how you can get the SSL certs in working state without having to set sslVerifyServerCert to false. 

🌟 Did this answer help you? If so, please consider:

  • Adding karma to show it was useful
  • Marking it as the solution if it resolved your issue
  • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

0 Karma

Namo
Explorer

@livehybrid 
I will check,Thanks

0 Karma

kiran_panchavat
Influencer

@Namo 

Could you please confirm the upgrade path — specifically, from which version to which version Splunk was upgraded?

Please note that you must first upgrade to the KV Store server version 4.2.x before proceeding with an upgrade to Splunk Enterprise 9.4.x or higher. For detailed instructions on updating to KV Store version 4.2.x (applicable to Splunk Enterprise versions 9.0.x through 9.3.x), refer to the official documentation: Migrate the KV store storage engine in the https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.3/administer-the-app-key-valu... 

We strongly recommend reviewing this guide to ensure a successful upgrade path and avoid issues like the one you're encountering.

 https://docs.splunk.com/Documentation/Splunk/9.3.2/Admin/MigrateKVstore

Did this help? If yes, please consider giving kudos, marking it as the solution, or commenting for clarification — your feedback keeps the community going!

Namo
Explorer

The KV sore was already on 4.2 before the upgrade.Hence went ahead with the splunk upgrade. 
The readme file was reviewed before the upgrade.
The error states 
Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:58948

Does it mean we need to review the certificate steps also.
Currently the server cert is issued by Digicert and cacert doesnt have Digicert as list of  trusted list

0 Karma

kiran_panchavat
Influencer

@Namo 

Please can you confirm if you followed the Splunk 9.4 upgrade pre-steps that are documented here?
https://docs.splunk.com/Documentation/Splunk/9.4.0/Installation/AboutupgradingREADTHISFIRST 
There is a section on upgrading the kv-store before running the Splunk 9.4 upgrade.

Reference:

https://splunk.my.site.com/customer/s/article/KV-store-status-failed-after-upgrade-to-9-4? 

Did this help? If yes, please consider giving kudos, marking it as the solution, or commenting for clarification — your feedback keeps the community going!

Namo
Explorer

Hi @kiran_panchavat 

There is no folder with the below name in the $SPLUNK_HOME/etc/apps folder

100_<stackname>_splunkcloud

0 Karma

tej57
Contributor

Hello @Namo,

There can be multiple reasons for KVStore failure. What version of Splunk did you upgrade from? Also, did you check the expiry of the certificate used by kvstore? Setting enabledSplunkdSSL to false will disconnect secure communication internally throughout the Splunk deployment wherever management port is being used. 

Thanks,
Tejas. 

LAME-Creations
Path Finder

We recently had the exact same issue on our environment.  It was all related to expiring certs.  It sounds like you have already looked into this, but I just want to echo that when the cert expires or can't be found, you can get this very behavior and it took a lot longer than I want to admit for our team to figure out that this was the reason all of our KV stores were failing.  

0 Karma

Namo
Explorer

Hello @tej57 
It was upgraded from 9.2 to 9.4. The cert is not expired ,is valid for another few days .
the  server cert is a combined cert

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...