https://community.splunk.com/t5/Splunk-Enterprise/KVStore-error-after-upgrade-to-9-0-1-Stand-alone-commands/m-p/621846#M14596 <P>I just had customer with similar issue on Windows HF.&nbsp;&nbsp;<BR />First there was the problem of expired cert.&nbsp; This was not just the&nbsp;</P><P>/opt/splunk/etc/auth/server.pem but also the pfx file in same location,<BR />And because of the way certs are managed on Windows in Splunk 9.x it was also necessary to delete via&nbsp;<BR />Windows Cert Management in Local / Personal / Certificates...Delete SplunkServerDefaultCert<BR /><BR />This got us past the expired cert issue...<BR />But then we found mongod.log detected Mongo version as "db version v4.2.19"....<BR />But then a few lines later complained about....</P><LI-CODE lang="markup">F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: BadValue: Invalid value for version, found 3.6, expected '4.2' or '4.0'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.6"</LI-CODE><P>&nbsp;<BR />So we had to cd to Splunk_Home/var/run/splunk/kvstore_upgrade<BR />Renamed versionFile42 to versionFile42.old<BR />Run the<A href="https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/MigrateKVstore#Migrate_manually_to_the_latest_version_of_Mongo_DB" target="_self"> standalone migration command again</A>...<BR /><SPAN>splunk migrate migrate-kvstore<BR /></SPAN>Start Splunk, KVStore working!<BR /><BR /><BR /><BR /></P> Wed, 23 Nov 2022 12:18:17 GMT dwest_splunk 2022-11-23T12:18:17Z https://community.splunk.com/t5/Splunk-Enterprise/KVStore-error-after-upgrade-to-9-0-1-Stand-alone-commands/m-p/617732#M14268 <P>I just went through this so posting here as I could not find the commands to fix it and had to open a ticket with support. Root cause was one of my servers had the server.pem certificate expired. This prevented the KV Store upgrade from working during the upgrade. This was on a windows heavy forwarder. Those are standalone, all the docs have commands for a SH cluster. So here are the standalone commands, hoping this helps someone. Took me a while to get them out of support.</P> <P>./splunk migrate kvstore-storage-engine --target-engine wiredTiger</P> <P>./splunk migrate migrate-kvstore</P> <P>./splunk show kvstore-status --verbose</P> <P>Thanks,</P> <P>Lee.</P> Wed, 23 Nov 2022 13:48:15 GMT https://community.splunk.com/t5/Splunk-Enterprise/KVStore-error-after-upgrade-to-9-0-1-Stand-alone-commands/m-p/617732#M14268 leekeener 2022-11-23T13:48:15Z https://community.splunk.com/t5/Splunk-Enterprise/KVStore-error-after-upgrade-to-9-0-1-Stand-alone-commands/m-p/621846#M14596 <P>I just had customer with similar issue on Windows HF.&nbsp;&nbsp;<BR />First there was the problem of expired cert.&nbsp; This was not just the&nbsp;</P><P>/opt/splunk/etc/auth/server.pem but also the pfx file in same location,<BR />And because of the way certs are managed on Windows in Splunk 9.x it was also necessary to delete via&nbsp;<BR />Windows Cert Management in Local / Personal / Certificates...Delete SplunkServerDefaultCert<BR /><BR />This got us past the expired cert issue...<BR />But then we found mongod.log detected Mongo version as "db version v4.2.19"....<BR />But then a few lines later complained about....</P><LI-CODE lang="markup">F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: BadValue: Invalid value for version, found 3.6, expected '4.2' or '4.0'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.6"</LI-CODE><P>&nbsp;<BR />So we had to cd to Splunk_Home/var/run/splunk/kvstore_upgrade<BR />Renamed versionFile42 to versionFile42.old<BR />Run the<A href="https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/MigrateKVstore#Migrate_manually_to_the_latest_version_of_Mongo_DB" target="_self"> standalone migration command again</A>...<BR /><SPAN>splunk migrate migrate-kvstore<BR /></SPAN>Start Splunk, KVStore working!<BR /><BR /><BR /><BR /></P> Wed, 23 Nov 2022 12:18:17 GMT https://community.splunk.com/t5/Splunk-Enterprise/KVStore-error-after-upgrade-to-9-0-1-Stand-alone-commands/m-p/621846#M14596 dwest_splunk 2022-11-23T12:18:17Z