Local KV store replication issues.

[nawazns5038]

Hi,

I am seeing some KV store replication errors on some of the search heads in the cluster. We wish to remove that search heads off the cluster as it has some host issues, is it safe to remove the hosts from the cluster if they have the following error.

They started appearing recently after we stopped and started some of the instances in the cluster.

Search peer  has the following message: Local KV Store has replication issues. See introspection data and mongod.log for details. Local instance has state Recovering.

And I am seeing this in the mongod.log

2019-03-15T23:27:57.682Z I QUERY [conn19] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:s_AGAMOTK@Pm97x3TDpqncvhmiDI7wvs_kv_ldaulqcNjuQ4964Y7C9ZSc23KqZ.system.profile query:{ $query: { ts: { $gte: new Date(1552611510438) }, millis: { $gte: 1000 } }, $orderby: {} }

I am seeing the same message in all the search heads which are having KV store errors.

Please help to resolve this issue. Or is it okay to remove the members out of the cluster ?

Thanks

[starcher]

You should open a support ticket. Get their assistance resolving the state of kvstore before removing cluster members. Additionally check the docs for your version of Splunk and make a backup of kvstore.

https://docs.splunk.com/Documentation/Splunk/7.2.4/Admin/BackupKVstore