Deployment Architecture

Search Peer down, distsearch.conf

SecurityFeller
Explorer

If I have 6 search peers configured in the distsearch.conf file but 3 of them go down, can Splunk recognize that a host is down and continue skipping down the list until it gets a live host?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, the on-prem search heads will be able to send queries to the AWS indexers.  Whether those queries are successful or not is another question the answer to which depends on how the indexers are configured.  Are they in a cluster?  What are the replication factor and search factor settings?

An indexer cluster with fully replicated and searchable data will be able to respond to search requests even if some peers are down.  The likelihood of the cluster being fully searchable goes down with each lost indexers.  If the indexers go down in rapid succession then it's possible (depending on the configuration) for some data to be unreachable.  In that case, the search requests will return incomplete results.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Unlike a forwarder sending data to a peer, search heads do not round-robin among the indexers.  Search queries are sent to all (most of the time) indexers and the responses are collated by the SH.  If the data on the 3 down peers is not replicated on the remaining 3 then you will get incomplete search results.

---
If this reply helps you, Karma would be appreciated.

SecurityFeller
Explorer

So:

  • if our search peers and indexers are synced across properly
  • Distconf has 6 IPs but only 3 of those hosts are up

Will our master search head cluster be able to still search against the peers? Or if it happens to hit a dead host it will return nothing for that query?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The Cluster Manager will keep track of where the searchable buckets are in the cluster.  If all goes well, you should be able to search with half the cluster still up.  It will depend on the search factor and the timing of the indexer failures as to whether the cluster will remain searchable.  The Indexer Clustering page on the Cluster Manager will tell you the state of the cluster.

---
If this reply helps you, Karma would be appreciated.

SecurityFeller
Explorer

I apologize, I don't believe my question was clear.

I have 2 full fledged splunk deployments, 1 on-prem and 1 in AWS.

The AWS SearchHeads are acting as remote search peers reside to the on-prem deployment. These search peers are hardcoded in the on-prem conf file as:

  • 10.0.0.1
  • 10.0.0.2
  • 10.0.0.3
  • 10.0.0.4
  • 10.0.0.5
  • 10.0.0.6

Now if the remote search peers 4-6 go down, will our on-prem splunk solution still be able to query our remote search peers as normal given that the config file has 3 non-live searchpeers

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, the on-prem search heads will be able to send queries to the AWS indexers.  Whether those queries are successful or not is another question the answer to which depends on how the indexers are configured.  Are they in a cluster?  What are the replication factor and search factor settings?

An indexer cluster with fully replicated and searchable data will be able to respond to search requests even if some peers are down.  The likelihood of the cluster being fully searchable goes down with each lost indexers.  If the indexers go down in rapid succession then it's possible (depending on the configuration) for some data to be unreachable.  In that case, the search requests will return incomplete results.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...