How can we handle under spec search heads?

[ddrillic]

Most of our Search Heads are of 252 GBs of RAM but there are some old VMs with 48 GBs of RAM. These ones have been unstable recently and one of them just crashed and needed a server reboot. Is there a way to delegate less work to such an under spec search head?

[inventsekar]

Hi @ddrillic as per reading other comments, let me assume that you are using SHC.
adding to Prakash007's idea, maybe, loadbalancing can help you i think.

i am not sure of this idea, but, let me suggest:
before the search heads, on the loadbalancing tier itself, you can configure it in such a way that, only part of the traffic/load/search requests will be sent to the SH's with less RAM Search Heads.

[ddrillic]

Interesting thinking @inventsekar - is anything like this supported by Splunk?

[richgalloway]

The SHC Captain assumes all search heads are the same when it's assigning scheduled searches. That can result in members with less memory or fewer cores getting overwhelmed with search jobs. The best solution is to allocate more memory to the old VMs, if possible, or upgrade them. Until then, @ddrillic's idea is a good one.

[ddrillic]

Thank you @richgalloway. @prakash007 idea makes lots of sense and the Cluster Administration class actually refers to it by saying that we can isolate scheduled searches, real-time searches and ad-hoc searches.

By the way @richgalloway, the plan is to remove these VMs from the SH cluster after the holidays.

[prakash007 · Accepted solution]

you can set the under spec search-head to run ad-hoc searches only...

http://docs.splunk.com/Documentation/Splunk/7.2.1/DistSearch/Adhocclustermember#Configure_a_member_t...

[ddrillic]

Very interesting @prakash007!