topic KVStore Failed 9.4.3 in Getting Data In

KVStore Failed 9.4.3

tech_g706 — Tue, 22 Jul 2025 16:14:47 GMT

Hi,
I upgraded Splunk Enterprise from 9.2.3 to 9.4.3, and the KVSotre status is failed.

It was migrated successfully to 7.0.14 on one server automatically, but on the second server, migration did not start upon upgrade.
Is there a solution to restore the KVstore status and migrate to 7.0.14?
It is a standalone server and not part of clustered environment.
Some servers also have KVStore status Failed on the version 9.2.3, and I want to change the status before starting to upgrade them to 9.4.3

This member:
backupRestoreStatus : Ready
disabled : 0
featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: 127.0.0.1:8191]
guid : xzy
port : 8191
standalone : 1
status : failed
storageEngine : wiredTiger
versionUpgradeInProgress : 0

Re: KVStore Failed 9.4.3

tech_g706 — Tue, 22 Jul 2025 17:53:07 GMT

Here are some internal logs:

2025-07-22T17:37:22.629Z I NETWORK [conn1078] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:43286 (connection id: 1078) 2025-07-22T17:37:22.629Z E NETWORK [conn1078] SSL peer certificate validation failed: self signed certificate in certificate chain
T 2025-07-22T17:37:22.125Z I NETWORK [conn1077] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:43272 (connection id: 1077)
h 2025-07-22T17:37:22.125Z E NETWORK [conn1077] SSL peer certificate validation failed: self signed certificate in certificate chain

Re: KVStore Failed 9.4.3

livehybrid — Tue, 22 Jul 2025 22:18:28 GMT

Hi @tech_g706

Do you have custom SSL Certs on your server?

Please can you confirm the output of the following which might help us dig down. Thanks

$SPLUNK_HOME/bin/splunk cmd btool server list --debug kvstore

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: KVStore Failed 9.4.3

tech_g706 — Wed, 23 Jul 2025 07:33:36 GMT

@livehybrid Thanks for the response.

Yes, some servers having custom certificates on those servers, we are having issue

If I try changing to the default local certificate, then it works

root@test02:/opt/splunk/bin# ./splunk cmd openssl verify -verbose -x509_strict -CAfile /opt/splunk/etc/auth/cacert.pem.default /opt/splunk/etc/auth/server.pem_old
/opt/splunk/etc/auth/server.pem_old: OK
root@test02:/opt/splunk/bin#
root@test02:/opt/splunk/bin#
root@test02:/opt/splunk/bin#
root@test02:/opt/splunk/bin# ./splunk cmd openssl verify -verbose -x509_strict -CAfile /opt/splunk/etc/auth/cacert.pem /opt/splunk/etc/auth/server.pem

error 20 at 0 depth lookup: unable to get local issuer certificate

./splunk cmd btool server list --debug kvstore
/opt/splunk/etc/system/default/server.conf [kvstore]
/opt/splunk/etc/system/default/server.conf clientConnectionPoolSize = 500
/opt/splunk/etc/system/default/server.conf clientConnectionTimeout = 10
/opt/splunk/etc/system/default/server.conf clientSocketTimeout = 300
/opt/splunk/etc/system/default/server.conf dbCursorOperationTimeout = 300
/opt/splunk/etc/system/default/server.conf dbPath = $SPLUNK_DB/kvstore
/opt/splunk/etc/system/default/server.conf defaultKVStoreType = local
/opt/splunk/etc/system/default/server.conf delayShutdownOnBackupRestoreInProgress = false
/opt/splunk/etc/system/default/server.conf disabled = false
/opt/splunk/etc/system/default/server.conf initAttempts = 300
/opt/splunk/etc/system/default/server.conf initialSyncMaxFetcherRestarts = 0
/opt/splunk/etc/system/default/server.conf kvstoreUpgradeCheckInterval = 5
/opt/splunk/etc/system/default/server.conf kvstoreUpgradeOnStartupDelay = 60
/opt/splunk/etc/system/default/server.conf kvstoreUpgradeOnStartupEnabled = true
/opt/splunk/etc/system/default/server.conf kvstoreUpgradeOnStartupRetries = 2
/opt/splunk/etc/system/default/server.conf minSnapshotHistoryWindow = 5
/opt/splunk/etc/system/default/server.conf oplogSize = 1000
/opt/splunk/etc/system/default/server.conf percRAMForCache = 15
/opt/splunk/etc/system/default/server.conf port = 8191
/opt/splunk/etc/system/default/server.conf replicaset = splunkrs
/opt/splunk/etc/system/default/server.conf replicationWriteTimeout = 1800
/opt/splunk/etc/system/default/server.conf shutdownTimeout = 100
/opt/splunk/etc/system/default/server.conf sslVerifyServerCert = false
/opt/splunk/etc/system/default/server.conf sslVerifyServerName = false
/opt/splunk/etc/system/default/server.conf storageEngine = wiredTiger
/opt/splunk/etc/system/default/server.conf storageEngineMigration = false

Re: KVStore Failed 9.4.3

thahir — Wed, 23 Jul 2025 15:26:42 GMT

Hi @tech_g706 ,

Sometime the issue with the MongoDB as well

Please check the following, it will helpful for further troubleshooting.

Mongodb status

ps -ef | grep -i mongod

if we are not getting any output means kvstore is not running.

check the below logs, try to find any clue on this logs

cat $SPLUNK_HOME/var/log/splunk/kvstore.log
cat $SPLUNK_HOME/var/log/splunk/mongod.log

Re: KVStore Failed 9.4.3

ahainline — Fri, 01 Aug 2025 16:46:22 GMT

We're experiencing the same issue. Were you able to resolve this?