KVStore error after upgrade to 9.0.1- Stand alone commands

[leekeener]

I just went through this so posting here as I could not find the commands to fix it and had to open a ticket with support. Root cause was one of my servers had the server.pem certificate expired. This prevented the KV Store upgrade from working during the upgrade. This was on a windows heavy forwarder. Those are standalone, all the docs have commands for a SH cluster. So here are the standalone commands, hoping this helps someone. Took me a while to get them out of support.

./splunk migrate kvstore-storage-engine --target-engine wiredTiger

./splunk migrate migrate-kvstore

./splunk show kvstore-status --verbose

Thanks,

Lee.

[dwest_splunk]

I just had customer with similar issue on Windows HF.  
First there was the problem of expired cert.  This was not just the 

/opt/splunk/etc/auth/server.pem but also the pfx file in same location,
And because of the way certs are managed on Windows in Splunk 9.x it was also necessary to delete via 
Windows Cert Management in Local / Personal / Certificates...Delete SplunkServerDefaultCert

This got us past the expired cert issue...
But then we found mongod.log detected Mongo version as "db version v4.2.19"....
But then a few lines later complained about....

F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: BadValue: Invalid value for version, found 3.6, expected '4.2' or '4.0'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.6"

 
So we had to cd to Splunk_Home/var/run/splunk/kvstore_upgrade
Renamed versionFile42 to versionFile42.old
Run the standalone migration command again...
splunk migrate migrate-kvstore
Start Splunk, KVStore working!