Splunk Enterprise

Why do I keep getting an error notification that a Searchhead is having trouble contacting kvstore cluster captain,[...]

briancronrath
Contributor

I keep getting an error message in our messages section at the top, stating that Search head cluster member ____ is having problems pulling configurations from the search head cluster captain.  Changes from the other members are not replicating to this member, and changes on this member are not replicating to other members. Consider performing a destructive configuration resync on this search head cluster member. 

 

I've performed destructive resync several times, and also when I use the show kvstore status function from any searchhead including the supposedly affected one it claims there are no actual issues with the searchhead, it just says it is a non-captain KV store member just like the rest.  If I try to manually close out the error message it just comes back.

What else can I look into here?

Labels (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

If you cannot get any useful information from internal logs, then maybe it's best to remove this node from SHC and then remove it's splunk installation and then reinstall, reinitialise and add it as a new node back to the SHC.

In most cases just clean local kvstore and then resync it will help. Of course if your time haven't sync in SHC then it could create this kind of issues?

Also check that your node's ssl certificate is valid and mongodb is running. Also check that those are in same level in all nodes. If I recall right in some 8.x version has newer version for it and if needed you must manually upgrade it on old nodes.

r. Ismo

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...