Is it possible to create a search head cluster where search heads have different hardware specs?

[dsmc_adv]

We are upgrading from Splunk 6.1 to 6.3, but the problem we are facing is that now is we are supporting a search head pool with 2 physical servers and 1 virtual with lower hardware. Since each server has its own config, it's not a big deal. We have the virtual out of the balancer and only use it through the API for some apps. With 6.3, if you want a cluster, the minimum number of nodes is 3, but I have read from the docs:

"Splunk recommends that you use homogeneous machines with identical hardware specifications for all cluster members. The reason is because the cluster captain assigns scheduled jobs to members based on their current job loads. When it does this, it does not have insight into the actual processing power of each member's machine. Instead, it assumes that each machine is provisioned equally."

Is there no way to create a search head cluster where a server has less weight than the others? Is only the scheduled jobs the main problem?

[somesoni2 · Accepted solution]

You should be able to get away with hon-homogeneous hardware setup, but would be better if you consider upgrading the hardware of your virtual node to match the 2 physical. To get around with load based scheduled jobs, you can change the scheduling mechanism from load based to round robin, bu setting following property in server.conf on SH cluster.

[shclustering]
scheduling_heuristic = <string>
* This setting configures the job distribution heuristic on the captain.
* There are currently two supported strategies: 'round_robin' or
  'scheduler_load_based'.
* Default is 'scheduler_load_based'.

Or you can set the virtual node to not to run scheduled searches by setting following property on server.conf on that node.

[shclustering]
adhoc_searchhead = <bool>
* This setting configures a member as an adhoc searchhead; i.e., the member
  will not run any scheduled jobs.
* Use the setting captain_is_adhoc_searchhead to reduce compute load on the
  captain.
* Defaults to false.

[dsmc_adv]

Adding more hardware is not an option at this moment. We are talking about servers with 32 cpu and 128gb of ram.

The other option was running with a deprecated search pool in 6.3. I will take a look of these parameters.

Thank you