Why does the master node, peer nodes, and search head have to be on its own instance?

[josegonzalezm]

We have a single server that is running indexer, master and search head. As we only have 1 server, it is a single point of failure.
We were thinking to put in place Splunk cluster solution, so our Splunk infrastructure would be resilient.
To deploy Splunk cluster we were thinking to use 2 servers:
* Server A: Indexer, master, search head.
* Server B: Indexer, master in standby, search head.

The documentation (https://docs.splunk.com/Documentation/Splunk/8.0.1/Indexer/Keydifferences) says "The master node, peer nodes, and search head must each run on its own instance."

Does anyone know why the 3 components have to be in different instances?

[gcusello]

Hi @josegonzalezm,
at first you have to decide if you want an HA architecture only for indexers or also for Search Heads.
The rules to implement an HA architecture are the following:

• Indexers, Search Heads and Master Node must be on different servers, so at least you need: 2 Indexers, 1 Search Head and 1 Master Node;
• you don't need to have two Master Nodes, only one;
• You need at least one Search Head because you cannot use Indexers and Master Node as Search Head;
• if you want also an HA architecture for Search Heads, you need at least 3 Search Heads (not less!) and a Deployer that can be located also on Master Node;
• Deployment Server must be located in a dedicated server if you have to manage more than 50 clients;
• you can find more infos at https://docs.splunk.com/Documentation/Splunk/8.0.1/Indexer/Clusterdeploymentoverview .

The reasons of this are:

• Master Node must manage Peers so cannot be located on one of them (there are addressing problems!);
• if you run a search on Indexers (in other words use one Indexer as Search Head) you don't have results (from version 7.0.0).

So summarizing:
you need at least 4 servers (2 Indexers, 1 Master Node, 1 Search Head) to have HA at data level, but not at search level (this means that dashboards, reports and alerts have a Single Point of Failure).
To have a full HA architecture, you need 6 or 7 servers:

• 2 Indexers,
• 1 Master Node and Deployer,
• 3 Search Heads
• and eventually 1 Deployment Server.

Ciao.
Giuseppe

[gfreitas]

Hey, your recommendation above is not supported neither recommended. The minimum number of instances to deploy Splunk on an Indexer Cluster will be 4 servers (2x indexers, 1x SH and 1x CM). Cluster Master is a relatively low resource server and you can use a smaller VM to accomplish that role. When you setup the Indexer Cluster and choose your instance type you can only chose one, that means you can't really have a CM and an Indexer together.
If you just have two physical hosts, you might want to virtualise them and create separate instances on those physical servers. Just make sure you have enough resources if virtualising. You can read this paper: https://www.splunk.com/pdfs/technical-briefs/splunk-deploying-vmware-tech-brief.pdf