Installation

Why receiving an ERROR when updating mmapv1 storage engine to wiredTiger?

kserverman
Explorer

I have a current single instance deployment of Splunk 8.2.3 on Linux Fedora 35, and it keeps encouraging me to update my mmapv1 storageEngine to wiredTiger. However, when I follow the instructions for a current single instance deployment at https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore#Migrate_the_KV_store_after_a... , it always fails after running the migration command. The entire output is:

Starting KV Store storage engine upgrade:
Phase 1 (dump) of 2:
.....ERROR: Failed to migrate to storage engine wiredTiger, reason=

where "reason" is blank. I haven't found anyone posting about getting this error without a reason. How should I complete the migration, or at least do further troubleshooting?

Labels (2)

dbray_sd
Path Finder

Same issue, same blank result, and same sad and unhelpful log files. Pretty much business as usual for Splunk. Not even sure why I'm surprised anymore. Miss the good ol' days when Splunk was worth the hefty price tag.

dsofoulis
Path Finder

I had the same issue. 

For me it was because I didnt have any data in my kvstore to migrate.

What worked for me was creating a kvstore and entering some dummy data. When I tried the migration again it was successful.

0 Karma

kserverman
Explorer

Unfortunately, my kvstore isn't empty. It's only about 733k, which isn't much, but I don't want to lose it in any case.

0 Karma

inventsekar
Super Champion

Hi @kserverman looks like there are lot of wiredTiger migration failures in these last few weeks. 

and as you know, the kvstore troubleshootings are complex as well. 

 

3 queries:

anything you find at the $SPLUNK_HOME/var/log/splunk/mongod.log ?!?!

do you use SSL certs(default/custom)?!?!

/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key file permissions please? is it 400?

PS ... If any post helped you in any way, pls give a hi-five to the author with an upvote. if your issue got resolved, please accept the reply as solution.. thanks.
0 Karma

kserverman
Explorer

Hi @inventsekar ,

I had upgraded Splunk to 8.2.4 earlier, and so tried the migration again to see if there's any change. Looking in $SPLUNK_HOME/var/log/splunk/mongod.log, I saw an error (included a couple lines before and after for context)

2022-01-03T01:10:30.062Z I ACCESS [conn1] Successfully authenticated as principal __system on local from client 127.0.0.1:58652
2022-01-03T01:10:30.062Z I NETWORK [conn1] end connection 127.0.0.1:58652 (0 connections now open)
mongodump fatal error: unrecognized DWARF version in .debug_info at 6
mongodump runtime stack:
mongodump panic during panic
mongodump runtime stack:
mongodump stack trace unavailable

2022-01-03T01:10:31.095Z I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2022-01-03T01:10:31.095Z I NETWORK [signalProcessingThread] shutdown: going to close listening sockets...
...

Looks like someone else found this error before: https://community.splunk.com/t5/Knowledge-Management/KV-Store-backup-migration-fail/m-p/573870#M8570 . I saw the same errors in splunkd.log as listed in this post.

Answering your other questions:

- I don't use SSL/TLS certs except for web access - these are valid custom ones, stored in $SPLUNK_HOME/splunkweb-certs/ .

-$SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key has 600 as permissions.

mjeanrichard
Explorer

Same here.

var/log/splunk/mongod.log shows the following:

mongodump linux-vdso.so.1 errno 13
mongodump fatal error: linux-vdso.so.1
mongodump runtime stack:
mongodump linux-vdso.so.1 errno 13
mongodump panic during panic
mongodump runtime stack:
mongodump linux-vdso.so.1 errno 13
mongodump stack trace unavailable

posnova
Engager

I also have the exact same error.

I have 1 search head and 4 indexers, not clustered.

Search Head and Indexer #1 migrated fine.  Indexers 2, 3, and 4 have this error.

 

0 Karma

kamaljagga
Path Finder

I am also getting same error.

Found this is in splunkd.log

ERROR KVStoreConfigurationProvider [63967 MainThread] - Failed to run mongodump, shutting down mongod 

0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...