Deployment Architecture

Search head cluster failure with 2 of 3 nodes - Can the user access Search head ?


I have 3 nodes Search Head Cluster, User access the single FQDN and my F5 load balancer share the load to these 3 search heads .
If 2 out of 3 search heads nodes failed what would be the expected outcome ? (as per the docs its mentioned the Entire Cluster fails) - but my F5 will still share the load to the alive node ... in this case,

  1. Will the user can still able to access the alive search head node (1 alive) in my cluster ? and what would happen to the user search request ?

from the docs, link to splunk doc
When a member fails,
If a search head cluster member fails for any reason and leaves the cluster unexpectedly, the cluster can usually continue to function without interruption: The cluster's high availability features ensure that the cluster can continue to function as long as a majority (at least 51%) of the members are still running. For example, if you have a cluster configured with seven members, the cluster will function as long as four or more members remain up. If a majority of members fail, the cluster cannot successfully elect a new captain, which results in failure of the entire cluster. See "Search head cluster captain."

0 Karma



Maybe the trick here is the statement "functioning cluster". If you don't have a majority, then no dynamic captain will be elected, so without a captain elected you don't have any scheduled searches being dispatched by the captain (as this is his job) to the other members. In that sense, the cluster stops functioning.

Still, if you manage to elect that single member as a static captain ( or in the case of the docs, any of the remaining members as a static captain), then that one will still dispatch scheduled searches to himself and if you allow so, still do ad-hoc searches.

If you have one search left in your cluster, your users will still search the data.

Let me know if this helps


Thanks . Yes, I am not interested in scheduled saved searches, I would need the users still can able to access the system and and do ad-hoc searches

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...