topic Re: Core Sizing Capacity in All Apps and Add-ons

Core Sizing Capacity

grivera_kudaw — Wed, 30 Sep 2020 00:39:42 GMT

Hi. Dears

Somebody can help me?

I need know what is the capacity of my Splunk to execute concurrents searches

This is currently the CPU capacity my servers

Server..........Physical_CPU......Vitual_CPU......CorexSocket......Socket......Thread x core......#CPUs
SH1.......................2..........................-.......................10.....................2.....................2.....................40
SH2 ......................1..........................-.........................8.....................1.....................2.......................6
SH3(virtual)..........-...........................2.........................6.....................2.....................1.....................12

IDX1......................2.........................-..........................4.....................2.....................2.....................16
IDX2......................2.........................-..........................8.....................2.....................1.....................16
IDX3(virtual).........-...........................2.........................8.....................2.....................1.....................16

Re: Core Sizing Capacity

bandit — Wed, 30 Sep 2020 00:42:29 GMT

The default will be base_max_searches(defaults to 6) + max_searches_per_cpu(defaults to 1) i.e. a 32-core host can run 38 concurrent searches out of the box.
The scheduler can use up to max_searches_perc (defaults to 60) i.e. up 23 of 38 total searches on a 32-core host, the rest being reserved for adhoc queries.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf

Btw, if you are running the monitoring console in distributed mode, it will show you the number of CPU Cores Splunk recognizes under the instances tab (The virtual number is what Splunk will use)

Example: CPU Cores (Physical / Virtual)
4 / 8

Re: Core Sizing Capacity

grivera_kudaw — Wed, 30 Sep 2020 00:42:45 GMT

Hi, Rob, thank you for your answer.

Then, to this case:
Considering 60% to scheduler searches

SH1 = ((6+48)*60)/100 = 32 Max concurrente searches
SH2 =((6+16)*60)/100 = 13 Max concurrente searches
SH3 = ((6+12)*60)/100 = 11 Max concurrente searches

The total to the Cluster is : 32+13+11= 56 concurrent searches that my system can
approximately execute?

Re: Core Sizing Capacity

grivera_kudaw — Fri, 24 May 2019 18:27:34 GMT

correction: SH1 = ((6+40)*60)/100 = 27 Max concurrente searches

Re: Core Sizing Capacity

skalliger — Fri, 24 May 2019 20:58:51 GMT

When I pressed "submit" I just saw your Monitoring Console paragraph. 😄 Deleted my comment.

Skalli

Re: Core Sizing Capacity

bandit — Wed, 30 Sep 2020 00:40:20 GMT

Yes, that looks valid for maximum searches than can be run by the Splunk scheduler. Note that you can adjust max_searches_per_cpu (although usually not a good idea). max_searches_perc could be adjusted up/down 10% etc. depending on whether the system will be running mostly scheduled searches/alerts vs. user adhoc searches.

Total Searches able to execute (scheduled and adhoc) would be:
SH1 = ((6+48) = 54 Max concurrente searches
SH2 =((6+16) = 22 Max concurrente searches
SH3 = ((6+12) = 18 Max concurrente searches

Re: Core Sizing Capacity

grivera_kudaw — Mon, 27 May 2019 15:23:20 GMT

Thanks for your help, Now I have it more clear

Re: Core Sizing Capacity

grivera_kudaw — Mon, 27 May 2019 15:24:52 GMT

hi, try again

Re: Core Sizing Capacity

grivera_kudaw — Wed, 30 Sep 2020 00:41:00 GMT

An latest question.

In the implemnattion I see this configuration by SH

limits.conf
max_historical_searches_per_cpu=4
base_max_searches = 6

then to this cluster will be:

max_historical_searches_per_cpu x number_of_cpus + base_max_searches
4*(40+16+12)+6 = 278
Or calculate one by one and after sum it?
4*40+6 =166
4*16+6= 70
4*12+6= 54
= 290
The results are differents

Well this configuration is wrong, but I dont have problems because the average of concurrent searches is the only 1.40

Re: Core Sizing Capacity

grivera_kudaw — Mon, 27 May 2019 17:02:39 GMT

And , I see an ancient message on the splunkd.log, say:

05-09-2019 13:46:11.904 -0400 ERROR SHCMaster - Search not executed: The maximum number of historical concurrent system-wide searches has been reached. current=108 maximum=105 for search: admin;xxxxxxxxx
But I don´t how obtained this value: 105 ??