How many resources do I commit to a master node in distributed multisite indexer clustering deployment?

[thomas_forbes]

I am in the process of setting up a distributed clustered deployment that spans 3 different sites. The deployment will live on virtual environment using VMware vSphere. I have determined the resource requirements for my indexers and search heads. I am having a little trouble figuring appropriate resources for the master node. Please help.

Thanks.

[lguinn2]

There are no published "minimum system requirements" for the cluster master. However, if you consider the functions of the cluster master, I think you can come up with some reasonable ideas:

The main job of the cluster master is to coordinate activities between indexers and search heads. It may also be contacted by forwarders if indexer discovery is configured. The cluster master is the repository of apps for the indexers. Clearly, the cluster master will need reliable high-speed networking. Since no data actually passes through the cluster master, disk I/O is not critical. This leads to the following suggestions:

1 - Network - the cluster master should have at least 1 GB NIC. The network segments between the cluster master, the indexers and the search heads should be low-latency and reliable. The overall network speed is one of the most critical components - not just the NIC configured for the cluster master.

2 - CPU - the cluster master will need sufficient CPU resources to drive the OS and Splunk, but also to support the network. You should examine the VMware specs to determine how much additional CPU to allocate for networking. I would guess that 4 cores at 2+ GHz would work, although it will depend on how many indexers/search heads and amount of indexing/searching. You can turn off hyperthreading, as it will not help your performance.

3 - Memory - Splunk recommends 12 GB memory for indexers; the cluster master should be able to use less. I would try 8 GB memory.

4 - Disk - for a cluster master, you need only enough disk for
a) the OS
b) Splunk
c) App repository for apps to be distributed to the indexers
There is no IOPS requirement for a cluster master, unlike an indexer.

For a physical machine, the hardware network card, motherboard bus bandwidth and the memory speed are important for network performance, but there isn't any way to deal with this in the virtual world except to say "make sure the underlying physical hardware is good enough."

You are probably aware of the issues of virtualizing Splunk (or any large data-intensive software). Here is a tech brief, just in case:
Deploying Splunk Enterprise® Inside Virtual Environments
The recommendations regarding CPU/memory reservations also apply to the cluster master. Raw volumes and high IOPS are not necessary for the cluster master. Don't forget to create a template for the cluster master (or a snapshot) that you can use to start a backup cluster master if needed.

You may need to tweak the configurations in the virtual environment to get the performance that you want for Splunk. Hopefully these suggestions will give you a starting point.

[thomas_forbes]

Thank you very much for your input.

[esix_splunk · Accepted solution]

What kind of data volumes are you planning for?

Typically the Master Node (Cluster Master) doesnt require heavy resources. In most use cases, a VM with 4gb / 8 core VM can handle a large deployment (30+ indexers.) This is mainly because its functionality is to distribute bucket tasks across the environment and announce to peers current availability.

Start with that as a baseline, and monitor resources. If you see high cpu / mem, then add more.

Now speaking of VMs. Is your indexing tier virtual also? You have to be very careful when using virtual indexers, in fact we typically do not recommend virtual indexers unless they are on dedicated storage or provisioned storage where IOPs are guaranteed. We need a minimum of 900iops per indexer. I mention this, because most virtual environments are using shared storage, typically a mix of local/DAS and NAS/SAN. And the indexing tier is always a problem if when the storage group says "You have 6000 iops available to that disk pool.." It means you have 6000 iops shared across N number of machines, where n > 10

Moral of the story.. be very careful on virtual indexers. Indexers need IOPs.. and when they dont have enough, the whole Splunk environment wont work.

[thomas_forbes]

Thank you for the quick response. I will let you know how everything went.