Installation

Why receiving an ERROR when updating mmapv1 storage engine to wiredTiger?

kserverman
Explorer

I have a current single instance deployment of Splunk 8.2.3 on Linux Fedora 35, and it keeps encouraging me to update my mmapv1 storageEngine to wiredTiger. However, when I follow the instructions for a current single instance deployment at https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore#Migrate_the_KV_store_after_a... , it always fails after running the migration command. The entire output is:

Starting KV Store storage engine upgrade:
Phase 1 (dump) of 2:
.....ERROR: Failed to migrate to storage engine wiredTiger, reason=

where "reason" is blank. I haven't found anyone posting about getting this error without a reason. How should I complete the migration, or at least do further troubleshooting?

Labels (2)

TassiloM
Engager

Had this issue in a pretty large cluster (over 40 Splunk servers).  KVStore Upgrade would not work on ANY host.

After lots of fiddling and trying to find a cause and solution, we had to give up. At least we found a reliable and easy enough workaround. Sharing it here in case someone else stumbles on this. This workaround is tested with v8.1.9, so should definitely work with versions above as well.

  1. splunk backup kvstore
  2. ls /opt/splunk/var/lib/kvstorebackup (tgz-Namen notieren)
  3. Edit /opt/splunk/etc/system/local/server.conf, add:
  4. [kvstore]
    storageEngine=wiredTiger
  5. systemctl shutdown Splunkd
  6. rm -r /opt/splunk/var/lib/kvstore
  7. systemctl start Splunkd
  8. splunk restore kvstore -archiveName <tgz-Name> 

Hope it helps some of you.

dbray_sd
Path Finder

Same issue, same blank result, and same sad and unhelpful log files. Pretty much business as usual for Splunk. Not even sure why I'm surprised anymore. Miss the good ol' days when Splunk was worth the hefty price tag.

dsofoulis
Path Finder

I had the same issue. 

For me it was because I didnt have any data in my kvstore to migrate.

What worked for me was creating a kvstore and entering some dummy data. When I tried the migration again it was successful.

0 Karma

kserverman
Explorer

Unfortunately, my kvstore isn't empty. It's only about 733k, which isn't much, but I don't want to lose it in any case.

0 Karma

inventsekar
Ultra Champion

Hi @kserverman looks like there are lot of wiredTiger migration failures in these last few weeks. 

and as you know, the kvstore troubleshootings are complex as well. 

 

3 queries:

anything you find at the $SPLUNK_HOME/var/log/splunk/mongod.log ?!?!

do you use SSL certs(default/custom)?!?!

/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key file permissions please? is it 400?

0 Karma

kserverman
Explorer

Hi @inventsekar ,

I had upgraded Splunk to 8.2.4 earlier, and so tried the migration again to see if there's any change. Looking in $SPLUNK_HOME/var/log/splunk/mongod.log, I saw an error (included a couple lines before and after for context)

2022-01-03T01:10:30.062Z I ACCESS [conn1] Successfully authenticated as principal __system on local from client 127.0.0.1:58652
2022-01-03T01:10:30.062Z I NETWORK [conn1] end connection 127.0.0.1:58652 (0 connections now open)
mongodump fatal error: unrecognized DWARF version in .debug_info at 6
mongodump runtime stack:
mongodump panic during panic
mongodump runtime stack:
mongodump stack trace unavailable

2022-01-03T01:10:31.095Z I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2022-01-03T01:10:31.095Z I NETWORK [signalProcessingThread] shutdown: going to close listening sockets...
...

Looks like someone else found this error before: https://community.splunk.com/t5/Knowledge-Management/KV-Store-backup-migration-fail/m-p/573870#M8570 . I saw the same errors in splunkd.log as listed in this post.

Answering your other questions:

- I don't use SSL/TLS certs except for web access - these are valid custom ones, stored in $SPLUNK_HOME/splunkweb-certs/ .

-$SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key has 600 as permissions.

mjeanrichard
Explorer

Same here.

var/log/splunk/mongod.log shows the following:

mongodump linux-vdso.so.1 errno 13
mongodump fatal error: linux-vdso.so.1
mongodump runtime stack:
mongodump linux-vdso.so.1 errno 13
mongodump panic during panic
mongodump runtime stack:
mongodump linux-vdso.so.1 errno 13
mongodump stack trace unavailable

posnova
Engager

I also have the exact same error.

I have 1 search head and 4 indexers, not clustered.

Search Head and Indexer #1 migrated fine.  Indexers 2, 3, and 4 have this error.

 

0 Karma

kamaljagga
Path Finder

I am also getting same error.

Found this is in splunkd.log

ERROR KVStoreConfigurationProvider [63967 MainThread] - Failed to run mongodump, shutting down mongod 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...