Knowledge Management

KV Store backup/migration fail

srondeau
New Member

I recently upgraded from 8.1 to 8.2.3, and noticed the message about migrating kvstore to wiredTIger. I decided to migrate, and followed the instructions here:

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore#Migrate_the_KV_store_after_a...

It failed because, I think, mongodump failed. The official reason in splunkd.log:

 

11-05-2021 14:10:57.695 -0700 ERROR MongodRunner [25826 MainThread] - MongtoolRunner exited with nonzero status=4
11-05-2021 14:10:57.695 -0700 ERROR KVStoreConfigurationProvider [25826 MainThread] - Failed to run mongodump, shutting down mongod

 

mongod.log output:

 

 

mongodump fatal error: unrecognized DWARF version in .debug_info at 6
mongodump runtime stack:
mongodump panic during panic
mongodump runtime stack:
mongodump stack trace unavailable

 

I removed the migration line in server.conf, started splunk, and tried to backup kvstore (both statuses were "ready") , and it failed to create anything in kvstorebackup; here is the relevant splunkd.log output:

 

11-05-2021 14:54:31.221 -0700 INFO KVStoreBackupRestore [27091 KVStoreBackupThread] - backup started for archiveName="kvdump_1636149271", using method=2
11-05-2021 14:54:31.284 -0700 ERROR MongodRunner [41130 BackupRestoreWorkerThread] - MongtoolRunner exited with nonzero status=4
11-05-2021 14:54:31.284 -0700 WARN KVStoreBulletinBoardManager [41130 BackupRestoreWorkerThread] - Failed to backup KV Store. Check for errors in the splunkd.log file in the $SPLUNK_HOME/var/log/splunk directory.

 

with the same mongodump errors as before.

Makes me think they are related. I checked my certificate (still good until 2024), permissions and ownerships, and all seem to be correct.

 

Any ideas?

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...