Reporting

How to calculate Splunk search concurrency limit for historical, scheduled, and real-time searches based on CPU cores?

Splunk Employee
Splunk Employee

I have often heard that Splunk's capability to run a certain number of concurrent searches depends on the CPU-core. Is it true? Also, do you have any guidelines on calculation search capacity based on CPU-core?

Splunk Employee
Splunk Employee

Limitation of Global Concurrent Searches
Number of "global" concurrent searches is limited by limits.conf configuration. For example, by default in v6.1.x, Splunk running on a four CPU-core server can run 10 concurrent "historical"(non-realtime) searches. This means that this search head can run only 10 max concurrent historical searches - the scheduler and summarization searches are a fraction of these 10. Keep in mind that scheduler/summarization searches are of a "lower priority" than user searches, for example if in this system you already had 10 concurrent user searches running, the scheduled searches would not be ran on time.

Limitation of concurrent searches in system Vs. user(per-role)
Each role defines number of concurrent searches for historical search and real-time search. As default, User role limits three concurrent historical searches and 6 real-time searches. Splunk instance wide limitation is defined by several attributes in limits.conf. Before 5.0, when the number of concurrent searches hit the limit, a new search was rejected and could not be run. In 5.0, Splunk keeps a new search in queue and wait for the number of concurrent searches go under the limit, instead of not running any new search job. Because of this behavior, default values for the attributes in limits.conf was changed.

  • limits.conf.spec [search] base_max_searches = 6 max_searches_per_cpu = 1 max_rt_search_multiplier = 1

[scheduler]
max_searches_perc = 50
auto_summary_perc = 50

  • authorize.conf.spec srchDiskQuota = 100 srchJobsQuota = 3 rtSrchJobsQuota = 6

Max Concurrent historical/real-time searches in 6.0

How Many general(historical) searches can be run concurrently?
If Splunk is running on a machine with 2 Cores per CPU x 2 (Total 4 CPU cores)
Attribute: max_searches_per_cpu = 1, base_max_searches = 6

max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches

10(max_hist_searches) = 1(max_search_per_cpu) x 4(number_of_cpus) + 6(base_max_searches)

How Many real-time searches can be run concurrently?
Attribute: 1x max_hist_searches ( max_rt_search_multiplier = 1 )

max real-time searches = max_rt_search_multiplier x max_hist_searches

10(max_realtime_searches) = 1(max_rt_search_multiplier) x 10(max_hist_searches)

How Many scheduled searches can be run concurrently?
Attribute: 50% of max historical searches (max_searches_perc = 50)

max scheduled historical searches = max_searches_perc x max_hist_searches

5(max_hist_scheduled_searches) = 0.5(max_searches_perc / 100) x 10(max_hist_searches)

max scheduled real-time searches = max_rt_search_multiplier x max l historical searches

5(max_realtime_scheduled_searches) = 0.5(max_searches_perc / 100) x 10(max_realtime_searches)

How Many Auto Summarization(Report Acceleration/DataModel Acceleration) searches can be run concurrently?

max scheduled real-time searches = max_realtime_scheduled_searches x auto_summary_perc(50%)

3 max_auto_summary_searches = 5 (max_hist_scheduled_searches) x 0.5 (auto_summary_perc / 100

Super Champion

on my system,
$SPLUNKHOME/etc/system/local/limits.conf
got this

the maximum number of concurrent searches per CPU

max_searches_per_cpu = 3

the base number of concurrent searches

base_max_searches = 10

max real-time searches = max_rt_search_multiplier x max historical searches

max_rt_search_multiplier = 1

on search head, Access controls » Roles » admin
mysplunk8000/en-US/manager/search/authorization/roles/admin?action=edit&uri=%2FservicesNS%2F-%2Fsearch%2Fauthorization%2Froles%2Fadmin

User-level concurrent search jobs limit = 50

now, the admin role will choose which option? 50 or 10?

0 Karma

Motivator

Is the number_of_cpus automaticall picked up by SPLUNK. Does it know that it is on a 4 core box, if not do we need to set 4 somewhere?
10(max_hist_searches) = 1(max_search_per_cpu) x 4(number_of_cpus) + 6(base_max_searches)

Builder

Hi @robertlynch2020,

If you were able to find an answer for this, can you please share it ?

Thanks,
Dev

0 Karma

Splunk Employee
Splunk Employee

Hi @damode,

It's set automatically to the number of logical processors when you first install Splunk. If you change the number of logical processors after installation, then the number will need to be adjusted manually (eg, if you assign more cores in your hypervisor or add an additional CPU in a physical server).

0 Karma

Engager

How would this calculation be handled in a Search Head Clustered environment? Do I consider each SH its own instance as far as CPU / Core count goes or do I multiply by the number of SHs i have in my cluster.

In my example I have 3 SHs clustered, Each SH has 8 cores (2 x CPUs, 4 Cores each CPU)

Thanks,
Greg

0 Karma

Splunk Employee
Splunk Employee

(Old question with an answer still valuable)

In a SHC the SH resources are not shared among searches, but for scheduled searches. Ad-hoc searches running on a SHC member will consider the cpu cores and memory resources as if it was unique. 

For the scheduler, the captain will mitigate starvation by distributing scheduled jobs to available members. Each member not being supposed to consume more than its capacity. So you give more resources to scheduled jobs automatically in a SHC,  while for adhoc searches it depends on where the users are logged in.

A load balancer for users connections can help, for high availability: users don't need to know each member fqdn nor ipod address to get in. 

0 Karma

Path Finder

Hi Greg,

Curious if you found an answer here. I'm also interested.

Thanks,

omg

0 Karma