Dedicated SH for Machine Learning Team

[jonaclough]

We have a 7 node SH multisite cluster, behind a VIP/LB. The ML team are coming up against limits for searches - some of which can be set on a user/search basis. However some limits are global (e.g. subsearch) and we cannot change these settings without risking platform instability.

The plan is to create a dedicated search head for the ML team. It would be part of the cluster but not behind the VIP. The ML team would get a separate GUI and REST VIP which would target the new SH.

The extra SH would mean an even number but I think this is OK if we ensure there is always an odd number in each site (5/3 in our case)

Does this sound like a sensible solution? 

[richgalloway]

Yes, that's sensible.  Keep in mind, however, that a SH that is part of a cluster is not truly "dedicated".  While only the ML may sign in to it, the SHC captain may choose to assign scheduled searches to any SHC member, including the ML node.

[jonaclough]

You can prevent the dispatch of adhoc jobs to a SH using below setting in server.conf. Technote

[shclustering]
adhoc_searchhead = true