Help with setting up Indexer Clustering

[neeravmathur]

Hi Guys,

Need some help with setting up Multisite Indexer Clustering. We have two DataCenters A&B. Below is the server architecture for these datacenters:

DATACENTER A

We have 3 Search Heads : SH-A,SH-B,SH-C (in a Search head cluster)

and we have 2 Indexers: IDX-1, IDX-2

DATACENTER B

We have 3 Disaster Recovery Search Heads: SH-A-DR,SH-B-DR,SH-C-DR (in a Search head cluster)

and 2 Indexers:IDX-3, IDX-4

Now, We want to setup Indexer clustering in such a way that

1. IDX—1 and IDX 3 are clustered
2. IDX-2 and IDX 4 are clustered

So that SH-A,B,C (in DC A) can search IDX-1 and IDX-2

While during DR

SH-A-DR,B-DR,C-DR (in DC B) can search IDX 3 and IDX 4.

What would be the best way to get this setup done?

Do we need to setup 2 Cluster Masters? If yes, then how to setup Search Head cluster with 2 Cluster Masters. Please suggest.

Thanks,

Neerav

[richgalloway]

You may be over-thinking it a bit.  You only need one cluster with all 4 indexers in it.  Set the site replication factor to ensure one copy of the data exists on each site.

site_replication_factor = origin:1,site1:1,site2:1,total:2
site_search_factor = origin:1,site1:1,site2:1,total:2

 Splunk automatically searches  the indexers in the local site.

[PickleRick]

+1 to what @richgalloway said. In general - you might be overthinking it. The whole point of multisite cluster is to ensure availability of data across sites and site replication factors and search factors are meant to ensure this according to your parameters.

Up until 8.x you had one cluster manager node (formerly cluster master) for the whole cluster. Since 9.0 I heard (haven't tested this yet) you may have a backup cluster manager.

But then again - you migh have some strange set of requirements that could result in creating two separate clusters (with that you must have one manager node per each cluster; you might have a backup manager so you might end up with having 4 managers for two two-node clusters; seems like an overkill).

[neeravmathur]

@PickleRick @richgalloway ,

Actually the site (DC) A is already up with IDX 1 & IDX 2 (NOT in Indexer Cluster) and so client has come up with this requirement. 

We are fine with having 2 different cluster masters (1 for each Indexer Cluster), but we have only active 1 search head cluster (In DC A) with DC B Search Head cluster being in DR mode.

So I guess the main question is there a way I can use these 2 Cluster Masters with 1 Search Head cluster?

Please suggest. Thanks for your help.

Thanks,

Neerav  

[PickleRick]

The question is what are the business requirements. Customer is not a Splunk architect so if customer tells you "we want two clusters organized this way" ask him "why?". And keep asking until he tells you his real needs. Having two clusters is not a business need. It's a technical issue/requirement. So if the customer needs simply to have sufficiently many copies of data, there's no problem with performing that on single cluster. So keep digging. I'm not saying that that can't be a use case where two separate clusters are indeed needed (for example - you must have separate environment for processing classified information) but it's relatively unlikely.

Oh, and remember that if you already have an environment with non-clustered indexers and have indexes on them existing buckets will not get converted to clustered ones and will not get replicated after you join the indexers in a cluster.

[neeravmathur]

@PickleRick @jamie00171 

Thanks for your suggestion. Discussion with the client are on... Will need to open firewall ports and test it out. Will keep you guys posted..

Thanks,

Neerav Mathur

[jamie00171 · Accepted solution]

Hi @neeravmathur 

Yes, it’s possible. It’s really just a case of configuring both cluster masters in server.conf of the SHs.

thanks,

jamie