I have 3 nodes Search Head Cluster, User access the single FQDN and my F5 load balancer share the load to these 3 search heads .
If 2 out of 3 search heads nodes failed what would be the expected outcome ? (as per the docs its mentioned the Entire Cluster fails) - but my F5 will still share the load to the alive node ... in this case,
from the docs, link to splunk doc
When a member fails,
If a search head cluster member fails for any reason and leaves the cluster unexpectedly, the cluster can usually continue to function without interruption: The cluster's high availability features ensure that the cluster can continue to function as long as a majority (at least 51%) of the members are still running. For example, if you have a cluster configured with seven members, the cluster will function as long as four or more members remain up. If a majority of members fail, the cluster cannot successfully elect a new captain, which results in failure of the entire cluster. See "Search head cluster captain."
Maybe the trick here is the statement "functioning cluster". If you don't have a majority, then no dynamic captain will be elected, so without a captain elected you don't have any scheduled searches being dispatched by the captain (as this is his job) to the other members. In that sense, the cluster stops functioning.
Still, if you manage to elect that single member as a static captain ( or in the case of the docs, any of the remaining members as a static captain), then that one will still dispatch scheduled searches to himself and if you allow so, still do ad-hoc searches.
If you have one search left in your cluster, your users will still search the data.
Let me know if this helps