Search head clustering with multisite indexing cluster - What happens when main site goes down?

[jofe]

Hi,

I'm designing a new Splunk solution based on Search head clustering on top of a multi site indexing cluster. (a small cluster that can grow)

Three search heads (search head cluster), four indexers, two sites. (2 site cluster)

Main data center : Two search heads and two indexers.
Remote data center : One search head and two indexers.

Master node and deployer is located on a VM in main site (can be moved to other site)

Search head config:
replication_factor=3 (all search heads should have complete set)
..
Index cluster config on master node.
[clustering]
mode = master
multisite=true
available_sites=site1,site2
site_replication_factor = origin:1,total:2 (Only one complete copy per data center)
site_search_factor = origin:1,total:2

Q1: Will this work, and is this a good idea? 😉
Q2: If main data center fails, will data still be searchable on remote site even if this search head can't be elected captain?
Q3: If this doesn't work, What must be done to the remote site to make it operational?

Thanks!

[matthieu_araman · Accepted solution]

I think it will work in the following mode
adhoc search (classic) OK
scheduled : disabled because no captain

but scheduling may be very important (and there are stuff done in the background which are scheduled)

I would certainly go for 2 SH on each site
It's active-active so takes this into account for sizing (it could be on a vm in some cases)

[mikaelbje]

Surprised you haven't received an official answer here. This is of great interest to a lot of folks. Did you figure out a working setup?