Knowledge Management

Why is KV Store status listed as failed after upgrade to version 7.2.5.1?

keio_splunk
Splunk Employee
Splunk Employee

KV Store is not starting up after upgrading Splunk to version 7.2.5.1.

./splunk show kvstore-status shows status as failed.
alt text

When performing KV Store migration, we are seeing the "App Key Value Store migration failed" error:

./splunk migrate migrate-kvstore

[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
WARN: [App Key Value Store migration] Service(34) terminated before the service availability check could complete. Exit code 2, waited for 2 seconds.
App Key Value Store migration failed, check the migration log for details. After you have addressed the cause of the service failure, run the migration again, otherwise App Key Value Store won’t function.

mongod.log reported the following error:
F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more detail

0 Karma
1 Solution

keio_splunk
Splunk Employee
Splunk Employee

The issue is caused by bug: SPL-167060 where the KV Store process fails to start up after upgrading to 7.2.x with IPv6 enabled. Bug: SPL-167060 has been resolved from Splunk version 7.2.6 onward.

Here are the steps to workaround the issue:
1. Stop Splunk.
2. Set server.conf:listenOnIPv6 = no.
3. Perform KVstore migration by executing: ./splunk migrate migrate-kvstore.
4. Set server.conf:listenOnIPv6 = yes.
5. Start Splunk.

View solution in original post

keio_splunk
Splunk Employee
Splunk Employee

The issue is caused by bug: SPL-167060 where the KV Store process fails to start up after upgrading to 7.2.x with IPv6 enabled. Bug: SPL-167060 has been resolved from Splunk version 7.2.6 onward.

Here are the steps to workaround the issue:
1. Stop Splunk.
2. Set server.conf:listenOnIPv6 = no.
3. Perform KVstore migration by executing: ./splunk migrate migrate-kvstore.
4. Set server.conf:listenOnIPv6 = yes.
5. Start Splunk.

Get Updates on the Splunk Community!

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...