Why does my KVStore replicationStatus show recovering?

[satyaallaparthi]

Hello,

I have a problem with my replication status.. getting the below result when ever I am trying to see kvstore status.

host 2 replication status is showing recovering from past 1 week.

I tried to resync both shcluster config using "splunk resync shcluster-replicated-config" and kvstore using "splunk resync kvstore -source MyGUID"

But, both didn't work.

Any help would be appreciated.

This member:
backupRestoreStatus : Ready
date : Mon Feb 3 15:50:19 2020
dateSec : 1580763019.583
disabled : 0
guid : ****-46ED-450B-BE21-******
oplogEndTimestamp : Mon Feb 3 15:50:18 2020
oplogEndTimestampSec : 1580763018
oplogStartTimestamp : Mon Feb 3 06:06:00 2020
oplogStartTimestampSec : 1580727960
port : 8191
replicaSet : splunkrs
replicationStatus : KV store captain
standalone : 0
status : ready

Enabled KV store members:
host1:8191
guid : ****-46ED-450B-BE21-******
hostAndPort : host1:8191
host2:8191
guid : ***-9A10-4962-9A3F-******
hostAndPort : host2:8191
host3:8191
guid : ****-F4B3-4E58-B765-*******
hostAndPort : host3:8191

KV store members:
host1:8191
configVersion : 205602
hostAndPort : host1:8191
lastHeartbeat : Mon Feb 3 15:50:17 2020
lastHeartbeatRecv : Mon Feb 3 15:50:17 2020
lastHeartbeatRecvSec : 1580763017.874
lastHeartbeatSec : 1580763017.873
optimeDate : Mon Feb 3 15:50:14 2020
optimeDateSec : 1580763014
pingMs : 0
replicationStatus : Non-captain KV store member
uptime : 253492

   host2:8191
                             configVersion : 205602
                               hostAndPort : host:8191
                             lastHeartbeat : Mon Feb  3 15:50:17 2020
                         lastHeartbeatRecv : Mon Feb  3 15:50:19 2020
                      lastHeartbeatRecvSec : 1580763019.366
                          lastHeartbeatSec : 1580763017.873
                                optimeDate : Wed Oct  2 11:09:46 2019
                             optimeDateSec : 1570028986
                                    pingMs : 0
                         replicationStatus : Recovering
                                    uptime : 253417

    host3:8191
                             configVersion : 205602
                              electionDate : Fri Jan 24 21:18:18 2020
                           electionDateSec : 1579918698
                               hostAndPort : host3:8191
                                optimeDate : Mon Feb  3 15:50:18 2020
                             optimeDateSec : 1580763018
                         replicationStatus : KV store captain
                                    uptime : 844587

Thanks,

[mustapha_arakji]

Your answer is in here: https://docs.splunk.com/Documentation/Splunk/9.0.3/Admin/ResyncKVstore

[jaihingorani]

if resyn does not work, you can try below, it has been working for me so far :
1. stop the splunk.
2../splunk clean kvstore --local
3. start splunk.

This triggers the initial Synchronization from other KVstore members

[humrish_b]

@satyaallaparthi  did you try the steps provided by @jaihingorani  was it successful. Could you let me know how this issue is resolved.