I have a single site cluster right now with the below configuration.
1. One License Server - also deployment server (001)
2. One Cluster Master (002)
3. Two Indexers (003, 004)
4. Two Heavy Forwarders (005, 006)
5. Two Search Servers (007, 008)
All the above servers are now opened for TCP 8089 between each other. There are about 100 Splunk Forwarders forwarding data to this 8 server cluster in Site1.
Here is the output of cluster-config on license master.
Now the plan is to setup another site - that'll also have about another 50 Splunk Forwarder that need to forward logs to the same set of indices. I have the below questions now.
1. I still need the 8 servers in the new Site2 - but the deployment server will be acting as a license slave. is that correct? I've already installed Splunk Enterprise same version (6.5.2) on about 7 servers and didn't do any configurations further.
2. Assuming servers 101 to 108 will be available in Site2 and the above configuration - what commands should I be executing to configure multisite clustering on all these 16 servers?
3. What values of searchfactor and replicationfactor should I consider?
4. For these servers to form a multisite cluster opening firewall ports between Site1 and Site2 - for splunk management and replication? How do I configure these replication ports?
5. I'd like to setup replication first and make sure that the logs of Site1 are searchable in Site2 search servers. Is that the right approach?
6. The Site2 Deployment Server is going to be a backup for Site1 Deployment Server or is going to be a new Deployment Server?
The end goal is to make the 4 search servers (2 from Site1 and 2 from Site2) be able to serve the same data - with 100 forwarders to Site1 and 50 forwarders to Site2.