After completing the upgrade from Splunk Enterprise version 9.3.3 to v9.4 the KVstore will no longer start. Splunk has yet to do the KVstore upgrade to v7 as the KVstore cannot start. We were already on 4.2 wiredtiger.
The problem we had, was our custom certificates did not have the proper extendedUsages set.
When we signed the certificates with extendedKeyUsage = serverAuth, clientAuth and restarted Splunk, the kvstore started, upgraded automatically and is running. It even works on search head clusters.
Note, the splunk documentation says that custom certificates are not working. But we've made it work
Here is the particular doc: https://docs.splunk.com/Documentation/Splunk/9.4.1/Admin/MigrateKVstore#Check_your_deployment
I am in the process of creating a supportcase with them.
Yay!
Here is how I figured out the issue:
Let's start the troubleshooting.
index=_internal log_level IN (warn, error) | chart count by component useother=falseSaw a lot of errors in components 'mongoclient' and 'KVstorageProvider'
Searching these components
index=_internal log_level IN (warn, error) component IN (KVStorageProvider, MongoClient)
04-08-2025 14:55:03.784 +0200 ERROR KVStorageProvider [37886 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '127.0.0.1:8191']
04-08-2025 14:55:04.370 +0200 WARN MongoClient [54380 KVStoreUpgradeStartupThread] - Disabling TLS hostname validation for localhost
Not very useful log messages. However, we can search the mongod.log as well
index=_internal source="/opt/splunk/var/log/splunk/mongod.log"
On my search head cluster peers, they had a very specific error in the field attr.error.errmsg:
(THIS will not show up on other splunk servers, but AS YOU WILL SEE, THIS IS THE ISSUE)
SSL peer certificate validation failed: unsupported certificate purpose
In this particular environment, we use custom certificates.
And to check what usages was allowed with my certificates, i ran the following command:
openssl x509 -in <path of my certificate> -noout -purpose
Notice that SSL server is Yes, whereas SSL client is No. Meaning this certificate is not able to be used for client authentication. GOTCHA!!!
So you need to create a new signing request, with an extendedKeyUsage
extendedKeyUsage = serverAuth, clientAuthHowever, it is up to the signer to actually respect this request. So I would double check after the CSR has been signed, that it has the correct extended purpose.
After pushing the new certificate to the server, and restarting Splunk, the kvstore automatically upgraded, and started after ~5 minutes.
I verified using this command:
/opt/splunk/bin/splunk show kvstore-status --verbose
Notice the serverVersion and uptime.
Good luck with the goddamn certificates. That was the solution for us
My kvstores were empty, but ideally one would try to search the kvstore in order to verify that it works.
Another way to verify is to check the monitoring console > Search > KV Store: Instance.
If you can see panels, the kvstore is working!
However, if the page is just white, it is not working 😞
My kvstores were empty, but ideally one would try to search the kvstore in order to verify that it works.
Another way to verify is to check the monitoring console > Search > KV Store: Instance.
If you can see panels, the kvstore is working!
However, if the page is just white, it is not working 😞
