How to upgrade Mongo in Splunk 9.0.0?

[amartin6]

https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/MigrateKVstore#Upgrade_KV_store_server_to_v...

Upgraded Splunk Enterprise version 9.0.0 from 8.2.5

Looking to see how to upgrade mongo from 4.0 to 4.2 on a single instance deployment.  During the Splunk Enterprise upgrade the migration to wiredTiger failed due to lack of disk space, the upgrade still continued and made the first hop of the mongo upgrade from version 3.6 to 4.0, it looks like after version 4.0 it tried to do the engine migration but couldn't because the lack of available disk space and therefore didn't do the last hop to version 4.2 of mongo.  We have since fixed the disk space issue and were able to complete the engine migration to wiredTiger, however don't know how to bump up the mongo version to 4.2.  The above link is for upgrading mongo in a cluster but not on a single instance, when looking at the options in splunk help kvstore I don't see anything for upgrading mongo either for a single instance, tried splunk start-shcluster-upgrade kvstore -version 4.2 -isDryRun true but of course it detected it wasn't a searchhead cluster.  Lastly trying to understand the difference in the output of mongo versionsbetween kvstore-status command versus splunk cmd mongod -version, clearly pulling from two different places.

[App Key Value Store migration] Starting migrate-kvstore.
Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile36
Started standalone KVStore update, start_time="2022-06-22 15:21:46".
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600seconds.
[App Key Value Store migration] Migration is not required.
Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile40
Not enough space to upgrade KVStore (or backup). You will need requiredBytes=3107201024 bytes, but KV Store DB filesystem only has availableBytes=2286272512
[App Key Value Store migration] Starting migrate-kvstore.
[App Key Value Store migration] Storage Engine hasn't been migrated to wireTiger. Cannot upgrade to service(42)

[splunk ~/var/run/splunk/kvstore_upgrade]$ splunk show kvstore-status --verbose |grep serverVersion
serverVersion : 4.0.24
[splunk ~/var/run/splunk/kvstore_upgrade]$

[splunk ~/var/run/splunk/kvstore_upgrade]$ splunk cmd mongod -version
db version v4.2.17-linux-splunk-v3
git version: be089838c55d33b6f6039c4219896ee4a3cd704f
OpenSSL version: OpenSSL 1.0.2zd-fips 15 Mar 2022
allocator: tcmalloc
modules: none
build environment:
distmod: rhel62
distarch: x86_64
target_arch: x86_64
[splunk ~/var/run/splunk/kvstore_upgrade]$

[amartin6 · Accepted solution]

We were able to upgrade mongo via the migrate command, thought it was only for the engine:

[splunk@ ~]$ splunk migrate migrate-kvstore
[App Key Value Store migration] Starting migrate-kvstore.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 2. This can take up to 600seconds.
[App Key Value Store migration] Migration is not required.
Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile42
Finished standalone KVStore update, stop_time="2022-06-29 13:41:29".
[splunk@ ~]$

[splunk@ ~]$ splunk show kvstore-status --verbose |grep serverVersion
serverVersion : 4.2.17
[splunk@ ~]$

[coreyCLI]

For anyone following/interested.  I had to perform the migrate kvstore commands twice to get to version 4.2.17.  Not sure why but the initial run gave the same output as the answer except this line:

"Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile40". 

Upon running it a second time I got the correct output:

"Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile42"

[haraksin]

It doesn't appear that the "serverVersion" value is available with this command even with the verbose tag in splunk versions 8.X

[amartin6]

Its a new field in splunk show kvstore-status --verbose but I don't recall what version of either the kvstore or Splunk that it was introduced

[haraksin]

Unfortunately the --verbose flag doesn't provide the serverVersion key/value pair either in 8.2.1

[amartin6]

Have you tried: splunk cmd mongod -version

[indreshdowjones]

Hi @amartin6

I am facing the same issue  and following are my versions.

./splunk cmd mongod -version
db version v4.2.17-linux-splunk-v4
git version: be089838c55d33b6f6039c4219896ee4a3cd704f
OpenSSL version: OpenSSL 1.0.2zf-fips 21 Jun 2022
allocator: tcmalloc
modules: none
build environment:
distmod: rhel62

distarch: x86_64
target_arch: x86_64

/splunk show kvstore-status --verbose |grep serverVersion
serverVersion : 4.0.24

I am getting the following message.

Failed to migrate KV Store storage engine to WiredTiger. This prevented running the latest version of KV Store. Migrate KV Store storage engine to WiredTiger and then upgrade KV Store. The Mmapv1 engine is deprecated. Learn more.

Starting migrate-kvstore.
[App Key Value Store migration] Storage Engine hasn't been migrated to wireTiger. Cannot upgrade to service(42)

Any idea what configuration i need to place in server.conf to fix this issue?[currently we are in splunk 9.0.3 version]

[amartin6]

We were prevented from migrating to the new engine: wiredTiger when we didn't have enough storage, once we cleaned up some disk space we were able to go back and run this after the upgrade to 9.x: splunk migrate migrate-kvstore (for standalone nodes), you'll get a message like this if you run it manually:

ERROR: Not enough space to upgrade KVStore, you will need requiredBytes=102776856576 bytes, but KV Store DB filesystem only has availableBytes=32339398656

We also completed the migration manually of some kvstores to the new wiredTiger engine but forgot to remove the storageEngineMigration=true line from the server.conf, also run a btool and make sure you don't have the engine hardcoded: splunk btool server list --debug |grep -i engine, wiredTiger is the default in 9.x

A helpful doc: https://docs.splunk.com/Documentation/Splunk/9.0.3/Admin/MigrateKVstore

[splunk@ ~]$ splunk show kvstore-status --verbose |grep -i engine
storageEngine : wiredTiger
[splunk@~]$

If the engine is set to mmapv1 it won't be able to upgrade to 4.2

[richgalloway]

You may want to contact support for assistance.

[indreshdowjones]

@richgalloway 

Any help here?