KVStore Migration to WiredTiger: Cluster Upgrade 8.0 > 8.2

[anirbandasdeb]

We are planning upgrade our clustered deployment (SH-C + IDX-C) from 8.0.5 to 8.2.1

In case we plan to migrate the KVStore engine to WiredTiger, do we have to mandatorily go through the v8.1 step as mentioned in https://docs.splunk.com/Documentation/Splunk/8.2.1/Admin/MigrateKVstore ? 

Per my understanding, the steps for this method stands like this

1. Upgrade cluster from 8.0.5 to 8.1.5 
2. Migrate the KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment
3. Upgrade cluster from 8.1.5 to 8.2.1

In case we decide not to use WiredTiger for now and complete the upgrade to 8.2.1, can we migrate the KV  Store to WiredTiger at a later point of time? 
To elaborate, will the following work: 

1. Upgrade cluster from 8.0.5 to 8.2.1 
2. At a later time, migrate the KV Store as instructed in Migrate the KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment

We do not use KV Store at all now, apart from whatever internal functions that Splunk Enterprise uses it for (no ITSI or ES as well), but want to plan ahead in case we use it in the future. 

Thanks in advance!

[anirbandasdeb]

We are planning upgrade our clustered deployment (SH-C + IDX-C) from 8.0.5 to 8.2.1

In case we plan to migrate the KVStore engine to WiredTiger, do we have to mandatorily go through the v8.1 step as mentioned in https://docs.splunk.com/Documentation/Splunk/8.2.1/Admin/MigrateKVstore ? 

Per my understanding, the steps for this method stands like this

1. Upgrade cluster from 8.0.5 to 8.1.5 
2. Migrate the KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment
3. Upgrade cluster from 8.1.5 to 8.2.1

In case we decide not to use WiredTiger for now and complete the upgrade to 8.2.1, can we migrate the KV  Store to WiredTiger at a later point of time? 
To elaborate, will the following work: 

1. Upgrade cluster from 8.0.5 to 8.2.1 
2. At a later time, migrate the KV Store as instructed in Migrate the KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment

We do not use KV Store at all now, apart from whatever internal functions that Splunk Enterprise uses it for (no ITSI or ES as well), but want to plan ahead in case we use it in the future. 

Thanks in advance!

[anirbandasdeb]

I just tested this on a standalone instance

1. Splunk Enterprise v8.0.5  
2. Set 
   

   %SPLUNK_HOME%\etc\system\local\server.conf
   [kvstore]
   storageEngineMigration=true

3. Upgrade to v8.2.1
4. Success! 

/opt/splunk/var/log/splunk/migration.log.2021-07-30: (only the relevant portion regarding kvstore migration)
=====================================================================
Starting KV Store storage engine upgrade:
Phase 1 (dump) of 2:
...............
Phase 2 (restore) of 2:
...................Data sanity check completed successfully for all configured KV Store collections
Successfully migrated to storage engine wiredTiger
=====================================================================


(dev2) splunk@splnkhfvm-3hqdt:~ $ ./bin/splunk btool --debug server list kvstore | grep local
/opt/splunk/etc/system/local/server.conf   [kvstore]
/opt/splunk/etc/system/local/server.conf   storageEngine = wiredTiger
/opt/splunk/etc/system/local/server.conf   storageEngineMigration = true

(dev2) splunk@splnkhfvm-3hqdt:~ $ ./bin/splunk show kvstore-status

 This member:
                           backupRestoreStatus : Ready
                                          date : Fri Jul 30 12:19:36 2021
                                       dateSec : 1627661976.923
                                      disabled : 0
                                          guid : 8297E92E-9E18-40DB-865C-54F7D9F82CBF
                             oplogEndTimestamp : Fri Jul 30 12:19:29 2021
                          oplogEndTimestampSec : 1627661969
                           oplogStartTimestamp : Fri Jul 30 11:35:07 2021
                        oplogStartTimestampSec : 1627659307
                                          port : 8191
                                    replicaSet : 8297E92E-9E18-40DB-865C-54F7D9F82CBF
                             replicationStatus : KV store captain
                                    standalone : 1
                                        status : ready
                                 storageEngine : wiredTiger

 KV store members:
        127.0.0.1:8191
                                 configVersion : 1
                                  electionDate : Fri Jul 30 11:35:07 2021
                               electionDateSec : 1627659307
                                   hostAndPort : 127.0.0.1:8191
                                    optimeDate : Fri Jul 30 12:19:29 2021
                                 optimeDateSec : 1627661969
                             replicationStatus : KV store captain
                                        uptime : 2675
​

So kvstore migration was a success when upgrading Splunk Enterprise from 8.0 to 8.2 in standalone mode. 

Next step: Validate if this is true with SH Cluster upgrade as well.