In the Monitoring Console, why does Search Head Clustering dashboard display "no captain available"?

[jmccallhbo]

In the Monitoring Console on 6.5, viewing the Search Head Clustering: Status and Configuration dashboard my SHC members are listed as up and have a consistent baseline but in the 'number of unpublished changes' column it says 'no captain is available'. Is this an error state? Can my SHC members not send changes to the SHC captain?

I had recently pushed a change from the deployer and the SHC members has an inconsistent baseline. So I then did a resync to fix the problem. Being that the resync is destructive I would like to not have to do that after every time I push configuration changes from the deployer.

[jmccallhbo]

I've resolved the issue I was having. When the monitoring console shows no captain available under unpublished changes it is indeed an error and means that the peers can not replicate changes to the captain. This can also be seen by running show shcluster-status and noticing that for the peers the last_conf_replication is in a state pending. Apparently the peers can still join the cluster even though they are unable to replicate changes with the master.

It turns out the problem I was having was networking related. Originally I had instances with only a private network interface. I later had to add a public interface to all my search heads and I believe this caused the problem because my firewall was only allowing traffic from the private ip addresses and I assume the default behavior of the search head cluster members was to replicate using the public interface. The solution I found was to set register_replication_address = <IP> in server.conf on the search heads which identifies the address used for receiving replication data.

[aaraneta_splunk]

Hi @jmcallhbo - Glad to hear that you found the solution to your question. Please don't forget to click "Accept" below your answer so that other users can easily find it, especially if they're having the same issue. Thank you!

[jmccallhbo]

Unfortunately I have not resolved the issue. The members of my cluster have begun to show "no captain available again" and I seem to be unable to recover. I think this may be related... Also, the version I'm using is 6.5.1.

RaftSimpleFileStorage - Error reading metadata1: error in raft mdata file for node version

[aaraneta_splunk]

No problem. We'll leave this question open to see if any other users want to try answering.