KV Store backup/migration fail

[srondeau]

I recently upgraded from 8.1 to 8.2.3, and noticed the message about migrating kvstore to wiredTIger. I decided to migrate, and followed the instructions here:

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore#Migrate_the_KV_store_after_a...

It failed because, I think, mongodump failed. The official reason in splunkd.log:

11-05-2021 14:10:57.695 -0700 ERROR MongodRunner [25826 MainThread] - MongtoolRunner exited with nonzero status=4
11-05-2021 14:10:57.695 -0700 ERROR KVStoreConfigurationProvider [25826 MainThread] - Failed to run mongodump, shutting down mongod

mongod.log output:

mongodump fatal error: unrecognized DWARF version in .debug_info at 6
mongodump runtime stack:
mongodump panic during panic
mongodump runtime stack:
mongodump stack trace unavailable

I removed the migration line in server.conf, started splunk, and tried to backup kvstore (both statuses were "ready") , and it failed to create anything in kvstorebackup; here is the relevant splunkd.log output:

11-05-2021 14:54:31.221 -0700 INFO KVStoreBackupRestore [27091 KVStoreBackupThread] - backup started for archiveName="kvdump_1636149271", using method=2
11-05-2021 14:54:31.284 -0700 ERROR MongodRunner [41130 BackupRestoreWorkerThread] - MongtoolRunner exited with nonzero status=4
11-05-2021 14:54:31.284 -0700 WARN KVStoreBulletinBoardManager [41130 BackupRestoreWorkerThread] - Failed to backup KV Store. Check for errors in the splunkd.log file in the $SPLUNK_HOME/var/log/splunk directory.

with the same mongodump errors as before.

Makes me think they are related. I checked my certificate (still good until 2024), permissions and ownerships, and all seem to be correct.

Any ideas?