my client is going to install Splunk Enterprise 7.1.3 on RHEL7.4 clusters
in the company's private cloud environment, and evaluating 7.2 Workload Management
seriously having the business demands of high and low priority search jobs on a
pair of Search Head and Indexer servers.
Before that, our IT vendor guieded us that we should have enough number of
CPU cores for long running search jobs, because the search jobs are almost tightly
bound with CPU cores. For example, if 8 searches of high
and 4 searches of low priority jobs concurrently at most, we would need 12
CPU cores for SH and more for IDX considering the search and injection processes.
Now at first, with Splunk 7.2 Workload Management, I consider the well-suited
resource allocation can be achieved with a smaller number of CPU cores, for ex,
80%_CPU_pool for high priority jobs and 20%_CPU_pool for low priority jobs with
only 10 CPU cores for the above 12 concurrent jobs at most.
(excluding other OS overhead processing requirements.)
Am I right?
Next, I am concerned about the following description of the doc.
Quote from Splunk Enterprise Workload Management 7.2.0 page 32:
If a search exceeds the maximum CPU resources allocated to its workload pool,
it is considered a soft limit, and the pool can borrow available CPU resources
from other pools.
In the above example of 8 high and 4 low priority jobs at most with 10 CPU cores
and 80%_CPU_pool(for high) and 20%_CPU_pool(for low), if 4 low priority jobs came
first and the CPU resources were borrowed from 80%_CPU_pool, and after that
8 high priority jobs would come in, how will the high priority jobs be served?
I'd expect CPU resource scheduling would rework quickly with small CPU cycle basis,
and the latter high priority jobs would be served with enough CPU cores quickly
over the low priority jobs. Am I right?