Installation

How to complete a KVStore migration?

ggvaca
Explorer

I have upgraded to Splunk 8.2.3 recently in a test environment. The last step in upgrading our slightly outdated environment is to migrate the KVStore. I don't understand the directions. What system are we going into "the server.conf in the $SPLUNK_HOME/etc/system/local/ directory." Is it in the Deployer? Cluster Master? Any of the search heads we have? If that file is edited in one system, how is it being pushed to the search heads? 

Also could we leave the variable storageEngineMigration=true in that file or does it have to stay as false when not migrating? 

Thank you!

@https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore

Labels (3)
Tags (1)
0 Karma
1 Solution

m_pham
Splunk Employee
Splunk Employee

I'm assuming you fall under this category in the documentation? -> https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore#Migrate_the_KV_store_after_a...

If so, it looks like you modify the "server.conf" on each of the search head members (step #1). Then just follow the other steps after that.

Probably doesn't need to be said but back up your KV store just in case...

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/BackupKVstore

View solution in original post

ggvaca
Explorer

What about leaving the variable storageEngineMigration=true in the server.conf? We use Git to store and run our files and I wasn't sure if this was a one and done situation after we upgrade or if we are going to want to push out this file to every system? It seems like this is a one and done once we upgrade. 

0 Karma

m_pham
Splunk Employee
Splunk Employee

I don't see anything about changing it back to false, so it seems like a permanent configuration.

 

storageEngineMigration = <boolean> 
* Whether or not you can migrate the KV Store storage engine on this instance.
* Migrating the storage engine means changing the engine from 'mmap' to 
  the newer 'wiredTiger'.
* If you set this to "true", the instance lets you migrate the engine,
  depending on the following scenarios:
  * If this instance is standalone, you can migrate the engine during
    an upgrade by enabling this setting with a "true" value as part of 
    the upgrade process and answering affirmatively when that process 
    prompts you, or after an upgrade by using the
    'splunk migrate kvstore-storage-engine' CLI command.
  * If it is a part of a search head cluster, you can perform the migration using the
    '/services/shcluster/captain/kvmigrate/start' REST endpoint.
* If you set this to "false", you cannot migrate the storage engine.
* Default: false

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/Serverconf

0 Karma

JustinSC
Explorer

I can attest that removing this stanza after the migration will cause Splunk to revert to the mmapv1 storage engine on restart. It will fail to start the kvstore and you cannot re-migrate the kvstore even if you add the setting back.

Now I have to figure out how to clean everything out and start over.

0 Karma

thambisetty
Super Champion

Splunk will migrate and upgrade kvstore automatically if the instance is running in standard alone mode. 

if Splunk instance is involved in any clustering ( search head , indexer ) then kvstore has to be manually migrated and upgraded. this applies to Deployer as well. 

there is no need of changing storageEngineMigration = true if you follow below link.

https://docs.splunk.com/Documentation/Splunk/9.0.2/Admin/MigrateKVstore?ref=hk

In search head cluster, its 3 step process.

 

#verify if kvstore can be migrated by running below command

splunk start-shcluster-migration kvstore -storageEngine wiredTiger -isDryRun true

#initiate migrate
/opt/splunk/bin/splunk start-shcluster-migration kvstore  -storageEngine wiredTiger 

#wait until it compltes, you can check status 
splunk show shcluster-kvmigration-status
#initiate upgrade 
splunk start-shcluster-upgrade kvstore -version 4.2
#wait until it completes, you can check status 
splunk show kvstore-status --verbose

output once it's completed
 KVStore Migration Status:
                migrationStatus: kvstore_migration_completed
                Start Time: Fri Nov  4 08:50:16 2022
                End Time: Fri Nov  4 08:55:06 2022
                Migration ID: 1667537416
                Peer Retries: 1
                Cluster Percentage: 100
                Storage Engine: wiredTiger

        Peer Status List:
                https://sh1:8089:
                        migrationStatus: succeeded
                https://sh2:8089:
                        migrationStatus: succeeded
                https://sh3:8089:
                        migrationStatus: succeeded


#finally remove unnecessary files 
Delete the following files from the $SPLUNK_HOME/bin directory:
mongod-3.6
mongod-4.0
mongodump-3.6
mongorestore-3.6

 

 

 

————————————
If this helps, give a like below.
0 Karma

m_pham
Splunk Employee
Splunk Employee

I'm assuming you fall under this category in the documentation? -> https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore#Migrate_the_KV_store_after_a...

If so, it looks like you modify the "server.conf" on each of the search head members (step #1). Then just follow the other steps after that.

Probably doesn't need to be said but back up your KV store just in case...

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/BackupKVstore

Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...