Why do I keep getting an error notification that a Searchhead is having trouble contacting kvstore cluster captain,[...]

[briancronrath]

I keep getting an error message in our messages section at the top, stating that Search head cluster member ____ is having problems pulling configurations from the search head cluster captain.  Changes from the other members are not replicating to this member, and changes on this member are not replicating to other members. Consider performing a destructive configuration resync on this search head cluster member. 

I've performed destructive resync several times, and also when I use the show kvstore status function from any searchhead including the supposedly affected one it claims there are no actual issues with the searchhead, it just says it is a non-captain KV store member just like the rest.  If I try to manually close out the error message it just comes back.

What else can I look into here?

[isoutamo]

If you cannot get any useful information from internal logs, then maybe it's best to remove this node from SHC and then remove it's splunk installation and then reinstall, reinitialise and add it as a new node back to the SHC.

In most cases just clean local kvstore and then resync it will help. Of course if your time haven't sync in SHC then it could create this kind of issues?

Also check that your node's ssl certificate is valid and mongodb is running. Also check that those are in same level in all nodes. If I recall right in some 8.x version has newer version for it and if needed you must manually upgrade it on old nodes.

r. Ismo