Why did KV store version upgrade failed?

[mika703]

Hi,

On two Deploymentservers i have the issue, that the KV Store Migration partly failes because the KV Store Version cant be upgraded.

"/opt/splunk/bin/splunk migrate migrate-kvstore" is interupted: 

[App Key Value Store migration] Starting migrate-kvstore.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 2. This can take up to 600seconds.
WARN: [App Key Value Store migration] Service(42) terminated before the service availability check could complete. Exit code 1, waited for 0 seconds.
App Key Value Store migration failed, check the migration log for details. After you have addressed the cause of the service failure, run the migration again, otherwise App Key Value Store won’t function.

In Splunkd Logs of the Server you afterwards find the following ERROR Message:

08-24-2022 15:57:45.252 +0200 ERROR KVStoreBulletinBoardManager [4946 MainThread] - Failed to upgrade KV Store to the latest version. KV Store is running an old version, service(40). Resolve upgrade errors and try to upgrade KV Store to the latest version again.

I assume it has something to do with the OS which is SUSE SLES 11.4, since on later RHEL Systems we dont run into the problem. Has somebody experienced the same?

[vgrote]

Hi,

same problem on SLES 12.

We upgraded to wiredTiger when we moved to 9.0.4 a while ago but only noticed the down rev version these days.

I opened a case with Splunk since the error message is not exactly helpful for me when the migration log is about a month old and refers to the migration log.

I will post the solution here.

VGVG

[vgrote]

On a side note: long after migrating to wiredTiger we stumbled over some version trouble after upgrading Splunk from 9.1.4 to 9.1.5.

It turned out that a simple "touch splunk/var/run/splunk/kvstore_upgrade/versionFile42" was able to resolve the problem.

[vgrote]

Correction: not on SLES12, one of the last remaining pockets with SLES11.

And the root cause looks rather simple:

$ splunk cmd mongod -version
mongod: /lib64/libpthread.so.0: version `GLIBC_2.12' not found (required by mongod)

$ ldd --version
ldd (GNU libc) 2.11.3

On SLES12 is comes back with

$ splunk cmd mongod -version
db version v4.2.17-linux-splunk-v4
...

$ ldd --version
ldd (GNU libc) 2.22

On the non-technical side this leads into the compatibility matrix and the various ways Linux distros are bundled and updated. Perhaps that library can be updated separately.

Hope this helps

Volkmar

[mika703]

thanks for the info, let me know👍

[isoutamo]

I haven’t seen this, but is there anything on mongodb.log? 

[mika703]

yes, i've checked the mongod.log but i can't see any log entries at the time i've tried to upgrade. Just one Message 5 minutes later, but i think thats not related to each other.

15:57:25 /opt/splunk/bin/splunk migrate migrate-kvstore

16:02:47.105Z I STORAGE [WT OplogTruncaterThread: local.oplog.rs] WiredTiger record store oplog truncation finished in: 0ms