Installation

Why did KV store version upgrade failed?

mika703
Engager

Hi,

On two Deploymentservers i have the issue, that the KV Store Migration partly failes because the KV Store Version cant be upgraded.

 

"/opt/splunk/bin/splunk migrate migrate-kvstore" is interupted: 

 

[App Key Value Store migration] Starting migrate-kvstore.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 2. This can take up to 600seconds.
WARN: [App Key Value Store migration] Service(42) terminated before the service availability check could complete. Exit code 1, waited for 0 seconds.
App Key Value Store migration failed, check the migration log for details. After you have addressed the cause of the service failure, run the migration again, otherwise App Key Value Store won’t function.

 

In Splunkd Logs of the Server you afterwards find the following ERROR Message:

08-24-2022 15:57:45.252 +0200 ERROR KVStoreBulletinBoardManager [4946 MainThread] - Failed to upgrade KV Store to the latest version. KV Store is running an old version, service(40). Resolve upgrade errors and try to upgrade KV Store to the latest version again.

 

I assume it has something to do with the OS which is SUSE SLES 11.4, since on later RHEL Systems we dont run into the problem. Has somebody experienced the same?

Labels (2)
0 Karma

vgrote
Path Finder

Hi,

same problem on SLES 12.

We upgraded to wiredTiger when we moved to 9.0.4 a while ago but only noticed the down rev version these days.

I opened a case with Splunk since the error message is not exactly helpful for me when the migration log is about a month old and refers to the migration log.

I will post the solution here.

VGVG

 

Tags (2)
0 Karma

vgrote
Path Finder

On a side note: long after migrating to wiredTiger we stumbled over some version trouble after upgrading Splunk from 9.1.4 to 9.1.5.

It turned out that a simple "touch splunk/var/run/splunk/kvstore_upgrade/versionFile42" was able to resolve the problem.

vgrote
Path Finder

Correction: not on SLES12, one of the last remaining pockets with SLES11.

And the root cause looks rather simple:

$ splunk cmd mongod -version
mongod: /lib64/libpthread.so.0: version `GLIBC_2.12' not found (required by mongod)

$ ldd --version
ldd (GNU libc) 2.11.3

On SLES12 is comes back with

$ splunk cmd mongod -version
db version v4.2.17-linux-splunk-v4
...

$ ldd --version
ldd (GNU libc) 2.22

On the non-technical side this leads into the compatibility matrix and the various ways Linux distros are bundled and updated. Perhaps that library can be updated separately.

Hope this helps

Volkmar

0 Karma

mika703
Engager

thanks for the info, let me know👍

0 Karma

isoutamo
SplunkTrust
SplunkTrust

I haven’t seen this, but is there anything on mongodb.log? 

0 Karma

mika703
Engager

yes, i've checked the mongod.log but i can't see any log entries at the time i've tried to upgrade. Just one Message 5 minutes later, but i think thats not related to each other.

 

15:57:25 /opt/splunk/bin/splunk migrate migrate-kvstore

16:02:47.105Z I STORAGE [WT OplogTruncaterThread: local.oplog.rs] WiredTiger record store oplog truncation finished in: 0ms

 

0 Karma
Get Updates on the Splunk Community!

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...

Splunk and Fraud

Watch Now!Watch an insightful webinar where we delve into the innovative approaches to solving fraud using the ...