SH's KVStore keeping "Initial Sync" status

[tlam_splunk]

Using Splunk v7.1.4 and find that one of SH is keeping "Initial Sync" in replication status of KVStore for few days.

Below logs in mongod.log:

"xxxx-xx-xxT06:58:52.058Z I QUERY [conn5] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:s_SplunkRXskOyCRWNcQsfE7@@TYWgMZ_WA_goaWbaVnkIQFzA7WcgP9a3MEXki.system.profile query:{ $query: { ts: { $gte: new Date(1552707830499) }, millis: { $gte: 1000 } }, $orderby: {} }"

Tried below 3 methods but it doesn't help

1.) splunk clean kvstore -local, restart splunk
2.) splunk clean kvstore -cluster, restart splunk
3.) splunk resync kvstore

[tlam_splunk]

Encounter a known issue on kvstore in v7.1.x. Splunk Search head cluster initial kv sync continues to fail.

Try to apply the following workaround and it helps.

Add the following attribute for all SH members and restart all SH members

server.conf
[kvstore]
allowUnsafeRenamesDuringInitialSync = true

NOTE: allowUnsafeRenamesDuringInitialSync attribute is only available in mongodb 3.4.10 and is not available prior to Splunk version 7.1.4.