topic Re: SHC Concurrency Search control in Splunk Search

SHC Concurrency Search control

0xAli — Thu, 25 Jun 2026 23:39:18 GMT

Hi,
I hope all is well.

I am writting to ensure i am getting the correct picture on the SHC search:

Assumption:
3x search head with CPU: 32 cores.
max search per cpu = 2

max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches (our environment is 70 per node)
Max Scheduled Search = 50% * max_hist_searches = 35
Max Summarization = 50% * Scheduled Search =17
Max Ad hoc guaranteed = 35 Remaining — always reserved for analysts

Only SCHEDULED searches are coordinated by the captain across nodes

Reference:
https://help.splunk.com/en/splunk-enterprise/administer/distributed-search/10.4/manage-search-head-…
https://community.splunk.com/t5/Reporting/How-to-calculate-Splunk-search-concurrency-limit-for-hist…
Are You Skipping? Please Read! | Splunk

For Real-time searches:
max_rt_searches = max_rt_search_multiplier x max_hist_searches
1. Max real-time searches = 1 * 70 = 70 ( our environment per node).
2. Max real-time scheduled searches = 50% x 70 = 35
3. Max Ad hoc guaranteed = 35 Remaining — always reserved for analysts

As per the above, we can conclude the below per each node:

Historical ad hoc → shared 70 pool (User manuall search past data)
Historical scheduled → 35 (50% of 70)
Summarization → 17 (50% of 35) (Datamodel)
RT scheduled → 35 (separate RT pool, independent)
RT ad hoc → 35 (RT guaranteed, reserved for analysts) (User manually search for RT live stream data)
Scheduled ad hoc → does not exist

Re: SHC Concurrency Search control

gcusello — Wed, 24 Jun 2026 06:29:45 GMT

Hi @0xAli ,

your settings seem to be correct, the only way to proceed is to analyze the load on your SHs using the Monitoring Consle, in this way you can see the health status of your system and eventually put some tuning action, e.g.:

check the scheduling of the scheduled searches,
avoid to use real-time searches and transforms them in scheduled searches,
optimize your custom searches,
check the performances of the Data Models,
etc...

Ciao.

Giuseppe

Re: SHC Concurrency Search control

0xAli — Wed, 24 Jun 2026 10:11:42 GMT

Thanks for your support, it's really appreciated!

Could you please clarify more on that point:

"avoid to use real-time searches and transforms them in scheduled searches,"

As from the calculated values, we have two separate pool, one for the historical scheduled search(35 - Summarization), and one for the RT-scheduled search(35), so the capacity of the RT will not affect the historical, and it's higher than historical (35 = Summarization + Historical scheduled search).

real-time searches meand ES detection + RT mode ?
scheduled searches means ES detection + Continous mode ?

Re: SHC Concurrency Search control

gcusello — Wed, 24 Jun 2026 13:48:19 GMT

Hi @0xAli ,

about first question:

in Splunk a search takes a CPU (more if you have also subsearches in the main search) until it will finish, so a RT search takes one or more CPUs for always, it's better to schedule a search so it will release the CPUs when finised.

About second question:

No RT depends on how you configured your detections: scheduled or RT, as I said if you can avoid RT, otherwise give more resources (CPUs to your Indexers and Search Heads)!

On Splunk Cloud RT searches are usually blocked for all not admin users!

Ciao.

Giuseppe

Re: SHC Concurrency Search control

0xAli — Wed, 24 Jun 2026 14:29:04 GMT

Now, i got your point, the confuse comes from the hint under the RT/scheduled mode

Re: SHC Concurrency Search control

gcusello — Wed, 24 Jun 2026 15:48:22 GMT

Hi @0xAli ,

is is clear for you or do you need other information?

Ciao.

Giuseppe

Re: SHC Concurrency Search control

PickleRick — Thu, 25 Jun 2026 22:08:01 GMT

Yes, the terminology here can be somewhat confusing. Splunk uses the term "real-time" in two different contexts when it comes to searching.

One thing is a real time search. It is a search which is done not against already indexed data but on the incoming data as it is ingested by Splunk. This type of search allocates one cpu on a search head running the search for the whole time the search is running and one cpu on each participating indexer. The other type of a search is a historical search which runs on data returned by indexers from the buckets residing on disk - already indexed data.

Another thing is the realtime schedule mode. This means that a run of a scheduled search will be attempted at the scheduled time but if there are no free search slots, Splunk will delay it for some time (if configuration of the search allows it) but if it still cannot find free slot to run it the search run will get skipped. The other scheduling mode is continuous which means that Splunk will try to run the search for a given time slot indefinitely until it finally can do so (I suppose there are some technical limits to that but that's the general idea). The caveat is that continuously scheduled searches have lower priority within the scheduler.

Re: SHC Concurrency Search control

0xAli — Thu, 25 Jun 2026 23:37:31 GMT

Thank you!

Let me leave what i have got from the truly insighful discussion with you all below:

1. Schedule Mode (Continous or real-time) doesn't determine the type of the Scheduled Search(RTScheduled Search OR Historical Scheduled Search ), it just determine the behaviour at the status: unavailable search slots, and how the Scheduled Search will run weather it will be skipped or DEFERRED?

then historical scheduled Search can be operate at the Continuous mode ot the Real-time mode ?
A Real-time scheduled Search always uses Real-time scheduling by nature?

If am i correct - as per the above - kindly, give me example on the Real-time scheduled Search and historical scheduled Search (when can i say it's RT-scheduled Search or historical scheduled Search)?

All event-based detection is historical scheduled search?

Thank in advance!

Re: SHC Concurrency Search control

gcusello — Fri, 26 Jun 2026 06:18:14 GMT

Hi @0xAli ,

about the first assumption, it isn't correct:

Schedule Mode (Continous or real-time) determine the type of the Scheduled Search(RTScheduled Search OR Historical Scheduled Search)

historical scheduled Search can be operate at the Continuous mode ot the Real-time mode ?

historical scheduled Search are in Continuous Mode, not in RT Mode, what's the sense of a RT historical search?

A Real-time scheduled Search always uses Real-time scheduling by nature?

yes

Ciao.

Giuseppe

Re: SHC Concurrency Search control

0xAli — Sat, 27 Jun 2026 13:18:22 GMT

Kindly, from the below screenshots, we can find that there are some RT scheduled search are enabled by default by splunk itself, and use the indexed real-time, so can we create some real-time scheduled search so we can accomodate the RT pool?

Re: SHC Concurrency Search control

PickleRick — Sat, 27 Jun 2026 13:24:56 GMT

No. If you turn on indexed real-time searches, it changes slightly the way real-time searches work. But it's still about real-time search, not about real-time schedule. (yes, I know it's confusing, I've already said so ;-))

Re: SHC Concurrency Search control

0xAli — Sat, 27 Jun 2026 13:26:46 GMT

Thank you, really appreciated!

Re: SHC Concurrency Search control

gcusello — Mon, 29 Jun 2026 07:02:53 GMT

Hi @0xAli ,

good for you, see next time!

let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

Re: SHC Concurrency Search control

0xAli — Tue, 30 Jun 2026 09:28:31 GMT

Guys,

Please read these articles carefully:

https://help.splunk.com/en/splunk-enterprise/search/search-manual/9.3/search-and-report-in-real-tim…
https://help.splunk.com/en/splunk-enterprise/administer/distributed-search/9.4/manage-search-head-c…
https://splunk.my.site.com/customer/s/article/Real-time-scheduling-vs-Continuous-scheduling

Summary:

the type of the scheduled search:
The Earliest and Latest time attributes are determine the type of the scheduled search weather it's Real-Time OR Historical

Rule:
Search for the past data = Historical -5m@m to -65m@m
Search for the Live stream = Real-Time rt-5m to rt-65m

the scheduling mode:
determine what is the behavior of the scheduled search when no available search slots, if it will be skipped or deferred.

Rule:
Continuous mode: Deferred
Real-Time mode: Skipped

topic Re: SHC Concurrency Search control in Splunk Search

SHC Concurrency Search control

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

All event-based detection is historical scheduled search?Thank in advance!

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

Re: SHC Concurrency Search control

All event-based detection is historical scheduled search?

Thank in advance!