Knowledge Management

Why is KV Store status listed as failed after upgrade to version 7.2.5.1?

keio_splunk
Splunk Employee
Splunk Employee

KV Store is not starting up after upgrading Splunk to version 7.2.5.1.

./splunk show kvstore-status shows status as failed.
alt text

When performing KV Store migration, we are seeing the "App Key Value Store migration failed" error:

./splunk migrate migrate-kvstore

[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
WARN: [App Key Value Store migration] Service(34) terminated before the service availability check could complete. Exit code 2, waited for 2 seconds.
App Key Value Store migration failed, check the migration log for details. After you have addressed the cause of the service failure, run the migration again, otherwise App Key Value Store won’t function.

mongod.log reported the following error:
F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more detail

0 Karma
1 Solution

keio_splunk
Splunk Employee
Splunk Employee

The issue is caused by bug: SPL-167060 where the KV Store process fails to start up after upgrading to 7.2.x with IPv6 enabled. Bug: SPL-167060 has been resolved from Splunk version 7.2.6 onward.

Here are the steps to workaround the issue:
1. Stop Splunk.
2. Set server.conf:listenOnIPv6 = no.
3. Perform KVstore migration by executing: ./splunk migrate migrate-kvstore.
4. Set server.conf:listenOnIPv6 = yes.
5. Start Splunk.

View solution in original post

keio_splunk
Splunk Employee
Splunk Employee

The issue is caused by bug: SPL-167060 where the KV Store process fails to start up after upgrading to 7.2.x with IPv6 enabled. Bug: SPL-167060 has been resolved from Splunk version 7.2.6 onward.

Here are the steps to workaround the issue:
1. Stop Splunk.
2. Set server.conf:listenOnIPv6 = no.
3. Perform KVstore migration by executing: ./splunk migrate migrate-kvstore.
4. Set server.conf:listenOnIPv6 = yes.
5. Start Splunk.

Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...