KV Store backup/migration fail

srondeau — Fri, 05 Nov 2021 22:19:11 GMT

I recently upgraded from 8.1 to 8.2.3, and noticed the message about migrating kvstore to wiredTIger. I decided to migrate, and followed the instructions here:

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore#Migrate_the_KV_store_after_an_upgrade_to_Splunk_Enterprise_8.1_or_higher_in_a_single-instance_deployment

It failed because, I think, mongodump failed. The official reason in splunkd.log:

11-05-2021 14:10:57.695 -0700 ERROR MongodRunner [25826 MainThread] - MongtoolRunner exited with nonzero status=4 11-05-2021 14:10:57.695 -0700 ERROR KVStoreConfigurationProvider [25826 MainThread] - Failed to run mongodump, shutting down mongod

mongod.log output:

mongodump fatal error: unrecognized DWARF version in .debug_info at 6 mongodump runtime stack: mongodump panic during panic mongodump runtime stack: mongodump stack trace unavailable

I removed the migration line in server.conf, started splunk, and tried to backup kvstore (both statuses were "ready") , and it failed to create anything in kvstorebackup; here is the relevant splunkd.log output:

11-05-2021 14:54:31.221 -0700 INFO KVStoreBackupRestore [27091 KVStoreBackupThread] - backup started for archiveName="kvdump_1636149271", using method=2 11-05-2021 14:54:31.284 -0700 ERROR MongodRunner [41130 BackupRestoreWorkerThread] - MongtoolRunner exited with nonzero status=4 11-05-2021 14:54:31.284 -0700 WARN KVStoreBulletinBoardManager [41130 BackupRestoreWorkerThread] - Failed to backup KV Store. Check for errors in the splunkd.log file in the $SPLUNK_HOME/var/log/splunk directory.

with the same mongodump errors as before.

Makes me think they are related. I checked my certificate (still good until 2024), permissions and ownerships, and all seem to be correct.

Any ideas?

topic KV Store backup/migration fail in Knowledge Management

KV Store backup/migration fail