KVStore Failed 9.4.3

[tech_g706]

Hi,
I upgraded Splunk Enterprise from 9.2.3 to 9.4.3, and the KVSotre status is failed.

It was migrated successfully to 7.0.14 on one server automatically, but on the second server, migration did not start upon upgrade. 
Is there a solution to restore the KVstore status and migrate to  7.0.14?
It is a standalone server and not part of clustered environment.
Some servers also have KVStore status Failed on the version 9.2.3, and I want to change the status before starting to upgrade them to 9.4.3


This member:
backupRestoreStatus : Ready
disabled : 0
featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: 127.0.0.1:8191]
guid : xzy
port : 8191
standalone : 1
status : failed
storageEngine : wiredTiger
versionUpgradeInProgress : 0

[thahir]

Hi @tech_g706 ,

Sometime the issue with the MongoDB as well 

Please check the following, it will helpful for further troubleshooting.

Mongodb status

ps -ef | grep -i mongod

if we are not getting any output means kvstore is not running.

check the below logs, try to find any clue on this logs

cat $SPLUNK_HOME/var/log/splunk/kvstore.log
cat $SPLUNK_HOME/var/log/splunk/mongod.log

[tech_g706]

@livehybrid  Thanks for the response.

Yes, some servers having custom certificates on those servers, we are having issue

If I try changing to the default local certificate, then it works

root@test02:/opt/splunk/bin# ./splunk cmd openssl verify -verbose -x509_strict -CAfile /opt/splunk/etc/auth/cacert.pem.default /opt/splunk/etc/auth/server.pem_old
/opt/splunk/etc/auth/server.pem_old: OK
root@test02:/opt/splunk/bin#
root@test02:/opt/splunk/bin#
root@test02:/opt/splunk/bin#
root@test02:/opt/splunk/bin# ./splunk cmd openssl verify -verbose -x509_strict -CAfile /opt/splunk/etc/auth/cacert.pem /opt/splunk/etc/auth/server.pem

error 20 at 0 depth lookup: unable to get local issuer certificate









./splunk cmd btool server list --debug kvstore
/opt/splunk/etc/system/default/server.conf [kvstore]
/opt/splunk/etc/system/default/server.conf clientConnectionPoolSize = 500
/opt/splunk/etc/system/default/server.conf clientConnectionTimeout = 10
/opt/splunk/etc/system/default/server.conf clientSocketTimeout = 300
/opt/splunk/etc/system/default/server.conf dbCursorOperationTimeout = 300
/opt/splunk/etc/system/default/server.conf dbPath = $SPLUNK_DB/kvstore
/opt/splunk/etc/system/default/server.conf defaultKVStoreType = local
/opt/splunk/etc/system/default/server.conf delayShutdownOnBackupRestoreInProgress = false
/opt/splunk/etc/system/default/server.conf disabled = false
/opt/splunk/etc/system/default/server.conf initAttempts = 300
/opt/splunk/etc/system/default/server.conf initialSyncMaxFetcherRestarts = 0
/opt/splunk/etc/system/default/server.conf kvstoreUpgradeCheckInterval = 5
/opt/splunk/etc/system/default/server.conf kvstoreUpgradeOnStartupDelay = 60
/opt/splunk/etc/system/default/server.conf kvstoreUpgradeOnStartupEnabled = true
/opt/splunk/etc/system/default/server.conf kvstoreUpgradeOnStartupRetries = 2
/opt/splunk/etc/system/default/server.conf minSnapshotHistoryWindow = 5
/opt/splunk/etc/system/default/server.conf oplogSize = 1000
/opt/splunk/etc/system/default/server.conf percRAMForCache = 15
/opt/splunk/etc/system/default/server.conf port = 8191
/opt/splunk/etc/system/default/server.conf replicaset = splunkrs
/opt/splunk/etc/system/default/server.conf replicationWriteTimeout = 1800
/opt/splunk/etc/system/default/server.conf shutdownTimeout = 100
/opt/splunk/etc/system/default/server.conf sslVerifyServerCert = false
/opt/splunk/etc/system/default/server.conf sslVerifyServerName = false
/opt/splunk/etc/system/default/server.conf storageEngine = wiredTiger
/opt/splunk/etc/system/default/server.conf storageEngineMigration = false

[tech_g706]

Here are some internal logs:


2025-07-22T17:37:22.629Z I  NETWORK  [conn1078] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:43286 (connection id: 1078) 2025-07-22T17:37:22.629Z E  NETWORK  [conn1078] SSL peer certificate validation failed: self signed certificate in certificate chain
T 2025-07-22T17:37:22.125Z I  NETWORK  [conn1077] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:43272 (connection id: 1077)
h 2025-07-22T17:37:22.125Z E  NETWORK  [conn1077] SSL peer certificate validation failed: self signed certificate in certificate chain

[ahainline]

We're experiencing the same issue.  Were you able to resolve this?

[livehybrid]

Hi @tech_g706 

Do you have custom SSL Certs on your server?

Please can you confirm the output of the following which might help us dig down. Thanks

$SPLUNK_HOME/bin/splunk cmd btool server list --debug kvstore

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing