Do I need a search-head in each cluster?

[a212830]

Hi,

I'm looking at the training material for the cluster admin course that I took, and for multi-site indexing, it says that the following are required:

1) One master
2) At least two indexers per site
3) At least one search-head per site

Why do I need a search-head at each site? I have 3 sites, but only intend to put SHC in two of the sites. I already have this setup, with SHP in one site, and multiple single-site clusters, and it works fine.

[sloshburch]

Is this all answered now? Just making sure after the discussion you are now all set.

[woodcock]

Search Head (peering to indexers) is the ultimate security: if a search head does not have an indexer as a search peer, then it cannot access the data. Install search heads as needed to cover documented maximum user/work load and also to partition access to indexers/data. It really doesn't matter where the search heads are because the size of a search (with the possible exception of Bundle Replication) is very small, and the size of the results coming back, is also usually pretty small.

[Runals]

In my experience you don't need a SH at each site. With multiple sites and replication in your environment though you will have to associate your SH with A site. If you don't your SH will not be able to differentiate between your data and will display all results (which could get ugly).

[lguinn2]

This is my experience as well. I like to say this: "indexers are committed to an indexer cluster, but search heads are only participants in the indexer cluster." If there is no search head in a particular location, the cluster master does not care. If there was a search head and it "went away" - the cluster master still doesn't care.

(The DMC will care that a Splunk instance became unreachable, but not the cluster master!)

Basically, after a search head first interacts with the cluster master and presents the cluster password, the cluster master allows the search head to make search requests. In a multi-site cluster, the search head should be assigned to a site; when the search head presents a request to the cluster master, the cluster master will return only indexers from the same site IF those indexers can fulfill the request. If the same-site indexers cannot fulfill the search request, the cluster master will give the search head a list of indexers from multiple sites.

[ChrisG]

You need a search head at each site that searches cluster data. If you want all searches to be local, you must install a search head at each site. See Search affinity in the Splexicon, and for full information, see Multisite indexer cluster deployment overview in the Managing Indexers and Clusters of Indexers manual.

[a212830]

I don't want all searches to be local. We do just the opposite - any replicated data is kept local.

[ChrisG]

But...search affinity is one of they key benefits of indexer clustering, because it reduces network traffic while you still get access to the entire set of data. Is there a reason you don't want to use it? Can you explain a little more about what you are trying to achieve?

[a212830]

It reduces network traffic on searches, but replicating data to other sites offsets that savings. We have very few indexes that require replication, and have decided that any data that is replicated will be replicated locally.

Our pre-6.3 architecture is 3 individual sites (regional), with only limited replicated indexes (all local), and two SH, using SHP. It works fine. I'll never replicate everything (or even the majority) of data, due to disk costs.

If the site goes down... it's down. Mgmt is aware of this, and willing to take the risk.

[ChrisG]

So it sounds as if you understand about search affinity, you have chosen not to use it, what you are doing is working for you, management understands the risks, and so your question is...what? I confess I am a little confused now.

[a212830]

Question was around the required SH in each site, which, sounds like is not really required. The training docs assume a certain setup, and mine doesn't match that setup. Just wanted to double-check...

[ChrisG]

Got it, thanks. The training docs assume a certain setup, as you say, and for a lot of reasons (search performance and availability) it's probably a best practice to have a search head at every site. But it will work the way you have it, and I don't see anything in the docs that directly contradict it. I also asked some smart people what they thought, and they couldn't think of any serious gotchas you're not already aware of.