topic Re: How to complete a KVStore Migration in Installation

How to complete a KVStore migration?

ggvaca — Tue, 22 Feb 2022 17:14:24 GMT

I have upgraded to Splunk 8.2.3 recently in a test environment. The last step in upgrading our slightly outdated environment is to migrate the KVStore. I don't understand the directions. What system are we going into "the server.conf in the $SPLUNK_HOME/etc/system/local/ directory." Is it in the Deployer? Cluster Master? Any of the search heads we have? If that file is edited in one system, how is it being pushed to the search heads?

Also could we leave the variable storageEngineMigration=true in that file or does it have to stay as false when not migrating?

Thank you!

@https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore

Re: KVStore Migration

m_pham — Tue, 22 Feb 2022 00:07:24 GMT

I'm assuming you fall under this category in the documentation? -> https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore#Migrate_the_KV_store_after_an_upgrade_to_Splunk_Enterprise_8.1_or_higher_in_a_clustered_deployment

If so, it looks like you modify the "server.conf" on each of the search head members (step #1). Then just follow the other steps after that.

Probably doesn't need to be said but back up your KV store just in case...

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/BackupKVstore

Re: How to complete a KVStore Migration

ggvaca — Tue, 22 Feb 2022 16:39:57 GMT

What about leaving the variable storageEngineMigration=true in the server.conf? We use Git to store and run our files and I wasn't sure if this was a one and done situation after we upgrade or if we are going to want to push out this file to every system? It seems like this is a one and done once we upgrade.

Re: How to complete a KVStore Migration

m_pham — Tue, 22 Feb 2022 17:02:49 GMT

I don't see anything about changing it back to false, so it seems like a permanent configuration.

storageEngineMigration = <boolean> * Whether or not you can migrate the KV Store storage engine on this instance. * Migrating the storage engine means changing the engine from 'mmap' to the newer 'wiredTiger'. * If you set this to "true", the instance lets you migrate the engine, depending on the following scenarios: * If this instance is standalone, you can migrate the engine during an upgrade by enabling this setting with a "true" value as part of the upgrade process and answering affirmatively when that process prompts you, or after an upgrade by using the 'splunk migrate kvstore-storage-engine' CLI command. * If it is a part of a search head cluster, you can perform the migration using the '/services/shcluster/captain/kvmigrate/start' REST endpoint. * If you set this to "false", you cannot migrate the storage engine. * Default: false

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/Serverconf

Re: How to complete a KVStore Migration

JustinSC — Tue, 26 Jul 2022 18:40:20 GMT

I can attest that removing this stanza after the migration will cause Splunk to revert to the mmapv1 storage engine on restart. It will fail to start the kvstore and you cannot re-migrate the kvstore even if you add the setting back.

Now I have to figure out how to clean everything out and start over.

Re: How to complete a KVStore Migration

thambisetty — Fri, 04 Nov 2022 05:40:52 GMT

Splunk will migrate and upgrade kvstore automatically if the instance is running in standard alone mode.

if Splunk instance is involved in any clustering ( search head , indexer ) then kvstore has to be manually migrated and upgraded. this applies to Deployer as well.

there is no need of changing storageEngineMigration = true if you follow below link.

https://docs.splunk.com/Documentation/Splunk/9.0.2/Admin/MigrateKVstore?ref=hk

In search head cluster, its 3 step process.

#verify if kvstore can be migrated by running below command splunk start-shcluster-migration kvstore -storageEngine wiredTiger -isDryRun true #initiate migrate /opt/splunk/bin/splunk start-shcluster-migration kvstore -storageEngine wiredTiger #wait until it compltes, you can check status splunk show shcluster-kvmigration-status #initiate upgrade splunk start-shcluster-upgrade kvstore -version 4.2 #wait until it completes, you can check status splunk show kvstore-status --verbose output once it's completed KVStore Migration Status: migrationStatus: kvstore_migration_completed Start Time: Fri Nov 4 08:50:16 2022 End Time: Fri Nov 4 08:55:06 2022 Migration ID: 1667537416 Peer Retries: 1 Cluster Percentage: 100 Storage Engine: wiredTiger Peer Status List: https://sh1:8089: migrationStatus: succeeded https://sh2:8089: migrationStatus: succeeded https://sh3:8089: migrationStatus: succeeded #finally remove unnecessary files Delete the following files from the $SPLUNK_HOME/bin directory: mongod-3.6 mongod-4.0 mongodump-3.6 mongorestore-3.6

Re: How to complete a KVStore Migration

computermathguy — Wed, 15 Feb 2023 15:15:48 GMT

Has anyone seen the following message after upgrading the storage engine to WireTiger?

Executing:
splunk migrate migrate-kvstore

Output:
[App Key Value Store migration] Collection data is not available.

Re: How to complete a KVStore Migration

vinothkumark — Thu, 16 Feb 2023 17:57:47 GMT

Do we need to initiate migration from SHC captain or KV Store captain?

Re: How to complete a KVStore Migration

Martin_Doering — Mon, 25 Sep 2023 16:35:06 GMT

Hi!

We are facing the same issue, both on versions 8.2.9 and version 8.2.12. We did not find a solution yet, but raised a support ticket.