Knowledge Management

Why is KV Store status listed as failed after upgrade to version 7.2.5.1?

keio_splunk
Splunk Employee
Splunk Employee

KV Store is not starting up after upgrading Splunk to version 7.2.5.1.

./splunk show kvstore-status shows status as failed.
alt text

When performing KV Store migration, we are seeing the "App Key Value Store migration failed" error:

./splunk migrate migrate-kvstore

[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
WARN: [App Key Value Store migration] Service(34) terminated before the service availability check could complete. Exit code 2, waited for 2 seconds.
App Key Value Store migration failed, check the migration log for details. After you have addressed the cause of the service failure, run the migration again, otherwise App Key Value Store won’t function.

mongod.log reported the following error:
F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more detail

0 Karma
1 Solution

keio_splunk
Splunk Employee
Splunk Employee

The issue is caused by bug: SPL-167060 where the KV Store process fails to start up after upgrading to 7.2.x with IPv6 enabled. Bug: SPL-167060 has been resolved from Splunk version 7.2.6 onward.

Here are the steps to workaround the issue:
1. Stop Splunk.
2. Set server.conf:listenOnIPv6 = no.
3. Perform KVstore migration by executing: ./splunk migrate migrate-kvstore.
4. Set server.conf:listenOnIPv6 = yes.
5. Start Splunk.

View solution in original post

keio_splunk
Splunk Employee
Splunk Employee

The issue is caused by bug: SPL-167060 where the KV Store process fails to start up after upgrading to 7.2.x with IPv6 enabled. Bug: SPL-167060 has been resolved from Splunk version 7.2.6 onward.

Here are the steps to workaround the issue:
1. Stop Splunk.
2. Set server.conf:listenOnIPv6 = no.
3. Perform KVstore migration by executing: ./splunk migrate migrate-kvstore.
4. Set server.conf:listenOnIPv6 = yes.
5. Start Splunk.

Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...