Deployment Architecture

Is it possible to revert the KV store storage engine migration in a standalone environment with SE 8.X?

ankitarath2011
Path Finder

Is it possible to revert the KV store storage engine migration in a standalone environment with SE 8.x. 

Example: If I am migrating the KV store storage engine from MMAP to WiredTiger.

Can I revert this change i.e. migrate from WireTiger to MMAP.  

If it is possible what are the steps to do so. Is there any doc for this?

I can see doc/command for migrating from MMAP to WiredTiger 

splunk migrate kvstore-storage-engine --target-engine wiredTiger

Need similar steps for the reverse condition.

Please help.

Labels (1)
0 Karma

yeahnah
Communicator

Assuming it was upgraded using storageEngineMigration=true in its server.conf file, here are the backout steps I've used to roll back to the mmapv1 MongoDB on a standalone Splunk instance.  It uses the backup of the old mmapv1 MongoDB taken at upgrade time.

NOTE: this could potentially cause data loss, if new data has gone into the migrated WiredTiger storage engine.  It really only makes sense if done soon after upgrade.

1. Stop Splunk Enterprise. Do not use the -f option.
2. Open server.conf in the $SPLUNK_HOME/etc/system/local/ directory
3. Edit the kvstore stanza and remove the following entries:
storageEngine = wiredTiger
storageEngineMigration=true

4. Save the server.conf file.
5. Change to the $SPLUNK_DB/kvstore directory
6. Rename the new mongo directory
mv mongo mongo_wiredtiger

7. Restore the previous mmpapv1 MongoDB under old_db directory
a) List the date stamped directory name
ls -ld old_db/*
b) Move and rename the old_db date stamped directory to mongo
mv old_db/<date stamped directory> mongo && ls -ld $SPLUNK_DB/kvstore/mongo

8. Restart splunk
splunk start

9. In the CLI, run the splunk show kvstore-status command
splunk show kvstore-status

10. This listed storage engine should be as follows
storageEngine : mmapv1


NOTE: this back out will not work after a SHC migration to use the Wired Tiger storage engine, as it does not take a snapshot (old_db) of the mmapv1 database at upgrade time.  It's not clear in the docs, but by the looks of it there is no easy way to revert a SHC back to the using mmapv1 after an upgrade.  If it all works then there is no good reason to revert to mmapv1.   Sadly, change control requirements usually require a documented backout plan.

Another possibility (I've not tried this) for a SHC restore to mmapv1 MongoDB could be:

  1. Stop all SHC members and take your own backup (could be compressed) of the $SPLUNK_DB/kvstore/mongo directory on all instances (ensure enough disk space)
  2. Restart SHC members and migrate to Wired Tiger, as per Splunk documentation, including a new backup of the current storage engine before starting
  3. RESTORE TO old mmav1 storage engine
  4. Stop all SHC members
    1. Rename mongoDB
      mv $SPLUNK_DB/kvstore/mongo $SPLUNK_DB/kvstore/mongo_wiredTiger
    2. Remove storageEngine = wiredTiger entry from server.conf in the $SPLUNK_HOME/etc/system/local/ directory
    3. Move the backed up mmapv1 back into place, e.g.
      mv $SPLUNK_DB/kvstore/mongo_mmapv1 $SPLUNK_DB/kvstore/mongo
  5. Restart SHC members one at a time and verify storage engine using CLI
      splunk show kvstore-status
  6. If Splunk SHC recovers OK, then you may also try this with the CLI (one SHC member only) to get to the latest backup
      splunk restore kvstore -archiveName <archive>

NOTE: understandably, the method listed above has no guaranteed consistency and would only ever be required in a disaster recovery scenario.

FINALLY:  As noted, I've not tried this on an SHC in my own environment (I was to late and had upgraded already) so validate it in your own test environments, if using. 

0 Karma

ankitarath2011
Path Finder

@bmunson_splunk Can you help on this?

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...