Knowledge Management

Why is KV Store status listed as failed after upgrade to version 7.2.5.1?

keio_splunk
Splunk Employee
Splunk Employee

KV Store is not starting up after upgrading Splunk to version 7.2.5.1.

./splunk show kvstore-status shows status as failed.
alt text

When performing KV Store migration, we are seeing the "App Key Value Store migration failed" error:

./splunk migrate migrate-kvstore

[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
WARN: [App Key Value Store migration] Service(34) terminated before the service availability check could complete. Exit code 2, waited for 2 seconds.
App Key Value Store migration failed, check the migration log for details. After you have addressed the cause of the service failure, run the migration again, otherwise App Key Value Store won’t function.

mongod.log reported the following error:
F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more detail

0 Karma
1 Solution

keio_splunk
Splunk Employee
Splunk Employee

The issue is caused by bug: SPL-167060 where the KV Store process fails to start up after upgrading to 7.2.x with IPv6 enabled. Bug: SPL-167060 has been resolved from Splunk version 7.2.6 onward.

Here are the steps to workaround the issue:
1. Stop Splunk.
2. Set server.conf:listenOnIPv6 = no.
3. Perform KVstore migration by executing: ./splunk migrate migrate-kvstore.
4. Set server.conf:listenOnIPv6 = yes.
5. Start Splunk.

View solution in original post

keio_splunk
Splunk Employee
Splunk Employee

The issue is caused by bug: SPL-167060 where the KV Store process fails to start up after upgrading to 7.2.x with IPv6 enabled. Bug: SPL-167060 has been resolved from Splunk version 7.2.6 onward.

Here are the steps to workaround the issue:
1. Stop Splunk.
2. Set server.conf:listenOnIPv6 = no.
3. Perform KVstore migration by executing: ./splunk migrate migrate-kvstore.
4. Set server.conf:listenOnIPv6 = yes.
5. Start Splunk.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...