I come back after sweating for hours on this, I could not go ahead with upgrading the entire Splunk Estate to version 9.4 before upgrading kvstore to version 7. First of all, thanks to @etoombs and @livehybrid for the guidance. Below is what helped me: 1. I believe kvstore upgraded the first time when i upgraded Splunk Enterprise to 9.4 on the Search Head, as i had "storageEngineMigration = true" under the [kvstore] stanza, but because of TLS issues, it could not load. 2. After upgrading to Splunk 9.4, i managed to bring kvstore up by setting "sslVerifyServerName = False" under the [kvstore] stanza, but it came back as version 4.2 3. To upgrade kvstore to version 7, i ran "./splunk start-standalone-upgrade kvstore -version 7.0 -dryRun true" and tailed logs from "/opt/splunk/var/log/splunk/splunkd.log" and all was well. 4. When i ran the actual kvstore upgrade command, i got the following error "KVStore Upgrade is not supported on this instance! Reason=Instance is configured with custom KVStore certificates". and once i applied "sslVerifyServerCert = False", and restarted splunk, kvstore came back as version 7 My current server.conf settings: [sslConfig] sslPassword = **************************************** sslVersions = tls1.2 serverCert = /opt/splunk/etc/auth/server.pem sslRootCAPath = /opt/splunk/etc/auth/cacert.pem sslVerifyServerName = False cliVerifyServerName = False [kvstore] serverCert = /opt/splunk/etc/auth/server.pem sslRootCAPath = /opt/splunk/etc/auth/cacert.pem sslVersions = tls1.2 sslPassword = **************************************** sslVerifyServerCert = False sslVerifyServerName = False storageEngine = wiredTiger #storageEngineMigration = true The below configs did not help me with anything, i was trying to get rid of this error " WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details" sslVerifyServerName = False cliVerifyServerName = False kvstore and MongoDB version featureCompatibilityVersion : 7.0 serverVersion : 7.0.14 db version v7.0.14 N.B: We are running default self-signed certs. For UI access we have 3rd party CA signed certs loaded on a reverse proxy. As a result, i did not mess with web.conf settings.
... View more
