Why is KV Store status listed as failed after upgrade to version 7.2.5.1?

[keio_splunk]

KV Store is not starting up after upgrading Splunk to version 7.2.5.1.

./splunk show kvstore-status shows status as failed.

When performing KV Store migration, we are seeing the "App Key Value Store migration failed" error:

./splunk migrate migrate-kvstore

[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
WARN: [App Key Value Store migration] Service(34) terminated before the service availability check could complete. Exit code 2, waited for 2 seconds.
App Key Value Store migration failed, check the migration log for details. After you have addressed the cause of the service failure, run the migration again, otherwise App Key Value Store won’t function.

mongod.log reported the following error:
F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more detail

[keio_splunk · Accepted solution]

The issue is caused by bug: SPL-167060 where the KV Store process fails to start up after upgrading to 7.2.x with IPv6 enabled. Bug: SPL-167060 has been resolved from Splunk version 7.2.6 onward.

Here are the steps to workaround the issue:
1. Stop Splunk.
2. Set server.conf:listenOnIPv6 = no.
3. Perform KVstore migration by executing: ./splunk migrate migrate-kvstore.
4. Set server.conf:listenOnIPv6 = yes.
5. Start Splunk.