Multi Site Index Cluster Deployment Suggestions

[akhan18]

Hello,

We are trying to deploy a multi-site cluster for high availability and disaster recovery process.  I looked at the Splunk valid architecture document and decided to move forward with M2/M12 deployment model with 2 indexers on each side instead of 3. I wanted to run my rational through some experts

Current Architecture 

• Search Head 
• 2 Indexers Configured for Distributed Search (12 CPU, 16GB RAM and 2TB) 
• 1 Heavy Forwarder that receives logs from a SysLog server 
• 70GB Daily License Usage (Plan on having 2 replicas and 1 searchable copy) 
• Version 8.1.1
• Ubuntu 18.04 latest patched
• No Deployment Server
• License Master and Monitoring Console

  I had a few questions 

  (i) What are the hardware requirements for the Splunk Indexers in a Clustered environment? 
  Note: Currently both Indexers are running on 12 CPU, 16GB RAM and 2TB Storage in VSphere. 
  I bumped that up to what Splunk recommends here along with 250GB Primary and10TB Secondary Disk 
  
  Mid-range indexer specification

  • An x86 64-bit chip architecture.
  • 24 physical CPU cores, or 48 vCPU at 2GHz or greater speed per core.
  • 64GB RAM.

     

    (ii) What are the hardware requirements for the Splunk Search Head? I am not planning on deploying a search head cluster and only a few users query Splunk. Is this okay?

    An x86 64-bit chip architecture.

    12 physical CPU cores. 

    32GB RAM.

    250 GB Primary Disk 
    
    

    (iii) I have Cluster Master that is on the Primary side that will manage the Index Cluster. I made this a separate server with decent specs. What are the recommended specs for a Cluster Master. I just hardened the server and add the Splunk Installer to it. Is there anything else I need to do before I configure the cluster?

    (iv) The existing indexers have disks that were partitioned using LVM so I can easily extend the 2TB to 10TB. How should I go about this with Splunk Indexers?  Any gotchas I have to look out for before lvextend -L +8T /dev/mapper ?

    (v) I have Universal forwarder and one heavy forwarder. Would I need to enable index discovery on both universal forwarder and the heavy forwarder in outputs.conf?
    
    (vi) I have two indexers on each side now that match the same specs. Can I join the cluster before the indexer discovery change or after? Are there any gotchas I should look out for before joining the cluster? I have backups and replication between both sites. I expect the traffic to be much higher between sites when replication is turned on. 
    
    (vii) We also have a KVStore, do I need to do something special for it when an Index Cluster is deployed?