Search Peer down, distsearch.conf

[SecurityFeller]

If I have 6 search peers configured in the distsearch.conf file but 3 of them go down, can Splunk recognize that a host is down and continue skipping down the list until it gets a live host?

[richgalloway]

Unlike a forwarder sending data to a peer, search heads do not round-robin among the indexers.  Search queries are sent to all (most of the time) indexers and the responses are collated by the SH.  If the data on the 3 down peers is not replicated on the remaining 3 then you will get incomplete search results.

[SecurityFeller]

So:

• if our search peers and indexers are synced across properly
• Distconf has 6 IPs but only 3 of those hosts are up

Will our master search head cluster be able to still search against the peers? Or if it happens to hit a dead host it will return nothing for that query?

[richgalloway]

The Cluster Manager will keep track of where the searchable buckets are in the cluster.  If all goes well, you should be able to search with half the cluster still up.  It will depend on the search factor and the timing of the indexer failures as to whether the cluster will remain searchable.  The Indexer Clustering page on the Cluster Manager will tell you the state of the cluster.

[SecurityFeller]

I apologize, I don't believe my question was clear.

I have 2 full fledged splunk deployments, 1 on-prem and 1 in AWS.

The AWS SearchHeads are acting as remote search peers reside to the on-prem deployment. These search peers are hardcoded in the on-prem conf file as:

• 10.0.0.1
• 10.0.0.2
• 10.0.0.3
• 10.0.0.4
• 10.0.0.5
• 10.0.0.6

Now if the remote search peers 4-6 go down, will our on-prem splunk solution still be able to query our remote search peers as normal given that the config file has 3 non-live searchpeers

[richgalloway · Accepted solution]

Yes, the on-prem search heads will be able to send queries to the AWS indexers.  Whether those queries are successful or not is another question the answer to which depends on how the indexers are configured.  Are they in a cluster?  What are the replication factor and search factor settings?

An indexer cluster with fully replicated and searchable data will be able to respond to search requests even if some peers are down.  The likelihood of the cluster being fully searchable goes down with each lost indexers.  If the indexers go down in rapid succession then it's possible (depending on the configuration) for some data to be unreachable.  In that case, the search requests will return incomplete results.