SH's KVStore keeping "Initial Sync" status

tlam_splunk — Tue, 29 Sep 2020 23:52:19 GMT

Using Splunk v7.1.4 and find that one of SH is keeping "Initial Sync" in replication status of KVStore for few days.

Below logs in mongod.log:

"xxxx-xx-xxT06:58:52.058Z I QUERY [conn5] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:s_SplunkRXskOyCRWNcQsfE7@@TYWgMZ_WA_goaWbaVnkIQFzA7WcgP9a3MEXki.system.profile query:{ $query: { ts: { $gte: new Date(1552707830499) }, millis: { $gte: 1000 } }, $orderby: {} }"

Tried below 3 methods but it doesn't help

1.) splunk clean kvstore -local, restart splunk
2.) splunk clean kvstore -cluster, restart splunk
3.) splunk resync kvstore

Re: SH's KVStore keeping "Initial Sync" status

tlam_splunk — Tue, 26 Mar 2019 08:07:11 GMT

Encounter a known issue on kvstore in v7.1.x. Splunk Search head cluster initial kv sync continues to fail.

Try to apply the following workaround and it helps.

Add the following attribute for all SH members and restart all SH members

server.conf
[kvstore]
allowUnsafeRenamesDuringInitialSync = true

NOTE: allowUnsafeRenamesDuringInitialSync attribute is only available in mongodb 3.4.10 and is not available prior to Splunk version 7.1.4.

topic SH's KVStore keeping "Initial Sync" status in Knowledge Management

SH's KVStore keeping "Initial Sync" status

Re: SH's KVStore keeping "Initial Sync" status