How to complete a KVStore migration?

[ggvaca]

I have upgraded to Splunk 8.2.3 recently in a test environment. The last step in upgrading our slightly outdated environment is to migrate the KVStore. I don't understand the directions. What system are we going into "the server.conf in the $SPLUNK_HOME/etc/system/local/ directory." Is it in the Deployer? Cluster Master? Any of the search heads we have? If that file is edited in one system, how is it being pushed to the search heads? 

Also could we leave the variable storageEngineMigration=true in that file or does it have to stay as false when not migrating? 

Thank you!

@https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore

[ggvaca]

What about leaving the variable storageEngineMigration=true in the server.conf? We use Git to store and run our files and I wasn't sure if this was a one and done situation after we upgrade or if we are going to want to push out this file to every system? It seems like this is a one and done once we upgrade. 

[m_pham]

I don't see anything about changing it back to false, so it seems like a permanent configuration.

storageEngineMigration = <boolean> 
* Whether or not you can migrate the KV Store storage engine on this instance.
* Migrating the storage engine means changing the engine from 'mmap' to 
  the newer 'wiredTiger'.
* If you set this to "true", the instance lets you migrate the engine,
  depending on the following scenarios:
  * If this instance is standalone, you can migrate the engine during
    an upgrade by enabling this setting with a "true" value as part of 
    the upgrade process and answering affirmatively when that process 
    prompts you, or after an upgrade by using the
    'splunk migrate kvstore-storage-engine' CLI command.
  * If it is a part of a search head cluster, you can perform the migration using the
    '/services/shcluster/captain/kvmigrate/start' REST endpoint.
* If you set this to "false", you cannot migrate the storage engine.
* Default: false

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/Serverconf

[JustinSC]

I can attest that removing this stanza after the migration will cause Splunk to revert to the mmapv1 storage engine on restart. It will fail to start the kvstore and you cannot re-migrate the kvstore even if you add the setting back.

Now I have to figure out how to clean everything out and start over.

[thambisetty]

Splunk will migrate and upgrade kvstore automatically if the instance is running in standard alone mode. 

if Splunk instance is involved in any clustering ( search head , indexer ) then kvstore has to be manually migrated and upgraded. this applies to Deployer as well. 

there is no need of changing storageEngineMigration = true if you follow below link.

https://docs.splunk.com/Documentation/Splunk/9.0.2/Admin/MigrateKVstore?ref=hk

In search head cluster, its 3 step process.

#verify if kvstore can be migrated by running below command

splunk start-shcluster-migration kvstore -storageEngine wiredTiger -isDryRun true

#initiate migrate
/opt/splunk/bin/splunk start-shcluster-migration kvstore  -storageEngine wiredTiger 

#wait until it compltes, you can check status 
splunk show shcluster-kvmigration-status
#initiate upgrade 
splunk start-shcluster-upgrade kvstore -version 4.2
#wait until it completes, you can check status 
splunk show kvstore-status --verbose

output once it's completed
 KVStore Migration Status:
                migrationStatus: kvstore_migration_completed
                Start Time: Fri Nov  4 08:50:16 2022
                End Time: Fri Nov  4 08:55:06 2022
                Migration ID: 1667537416
                Peer Retries: 1
                Cluster Percentage: 100
                Storage Engine: wiredTiger

        Peer Status List:
                https://sh1:8089:
                        migrationStatus: succeeded
                https://sh2:8089:
                        migrationStatus: succeeded
                https://sh3:8089:
                        migrationStatus: succeeded


#finally remove unnecessary files 
Delete the following files from the $SPLUNK_HOME/bin directory:
mongod-3.6
mongod-4.0
mongodump-3.6
mongorestore-3.6

[vinothkumark]

Do we need to initiate migration from SHC captain or KV Store captain?

[computermathguy]

Has anyone seen the following message after upgrading the storage engine to WireTiger?

Executing:
splunk migrate migrate-kvstore

Output:
[App Key Value Store migration] Collection data is not available.

[Martin_Doering]

Hi!

We are facing the same issue, both on versions 8.2.9 and version 8.2.12. We did not find a solution yet, but raised a support ticket.

[m_pham · Accepted solution]

I'm assuming you fall under this category in the documentation? -> https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore#Migrate_the_KV_store_after_a...

If so, it looks like you modify the "server.conf" on each of the search head members (step #1). Then just follow the other steps after that.

Probably doesn't need to be said but back up your KV store just in case...

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/BackupKVstore