Knowledge Management

Local KV store replication issues.

nawazns5038
Builder

Hi,

I am seeing some KV store replication errors on some of the search heads in the cluster. We wish to remove that search heads off the cluster as it has some host issues, is it safe to remove the hosts from the cluster if they have the following error.

They started appearing recently after we stopped and started some of the instances in the cluster.

Search peer  has the following message: Local KV Store has replication issues. See introspection data and mongod.log for details. Local instance has state Recovering.

And I am seeing this in the mongod.log

2019-03-15T23:27:57.682Z I QUERY [conn19] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:s_AGAMOTK@Pm97x3TDpqncvhmiDI7wvs_kv_ldaulqcNjuQ4964Y7C9ZSc23KqZ.system.profile query:{ $query: { ts: { $gte: new Date(1552611510438) }, millis: { $gte: 1000 } }, $orderby: {} }

I am seeing the same message in all the search heads which are having KV store errors.

Please help to resolve this issue. Or is it okay to remove the members out of the cluster ?

Thanks

0 Karma

starcher
SplunkTrust
SplunkTrust

You should open a support ticket. Get their assistance resolving the state of kvstore before removing cluster members. Additionally check the docs for your version of Splunk and make a backup of kvstore.

https://docs.splunk.com/Documentation/Splunk/7.2.4/Admin/BackupKVstore

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...