Knowledge Management

How to resolve error on a search head member in the search head cluster: "Local KV Store has replication issues...Local instance has state Recovering."?

kranitha
Explorer

Hi All,

One of the search head members in the search head cluster has a message: "Local KV Store has replication issues. See introspection data and mongod.log for details. Local instance has state Recovering..".

What can I do to fix the issue?
When I checked with the kvstore status command for this particular SH member, the status is shown as recovering.
On using the resync command, even then the issue still exists.

Can you please let me know what steps should be followed to rectify the issue.
Will there be any impact on the performance of search heads?

Labels (1)
0 Karma
1 Solution

kranitha
Explorer

Fixed the issue by following below steps:
1. Login to the SHM and navigate to the bin directory . Take a backup of KVstore using the below command
./splunk backup kvstore

  1. Now Clean the kvstore locally by stopping the service ./splunk stop ./splunk clean kvstore –local ----- Accept the scary message as "Y" ./splunk start

View solution in original post

kranitha
Explorer

Fixed the issue by following below steps:
1. Login to the SHM and navigate to the bin directory . Take a backup of KVstore using the below command
./splunk backup kvstore

  1. Now Clean the kvstore locally by stopping the service ./splunk stop ./splunk clean kvstore –local ----- Accept the scary message as "Y" ./splunk start

mustapha_arakji
Splunk Employee
Splunk Employee
0 Karma

denissotoacc
Path Finder

When you said "SHM" is it the Deployer, the Search Head Captain or the same SH with the issue?

0 Karma

lenrigodoy
Explorer

The same message appears me today over one of the SHC members. To solve, I follow the recommendations over the member (I have 3 SH, the last one was the problematic). 

First of all, you need to backup the KVstore. Splunk service should be running over the SH to backup.

Later, you need to restore the KVStore. Splunk service should be stopped before of the command execution.

All of these commands should be executed over the affected member. If you have multiple members affected, you would execute it on each affected member, or in all SH if all are being affected with this replication error.

Last Splunk version (8.2.4) recommends to move KVStore to a new technology. Currently, Splunk uses mmapv1 as Storage Engine, but it's recommended to move to wiredTiger. Follow the docs to migrate your KVStore to recommended engine:

Migrate the KV store storage engine - Splunk Documentation

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...