Did 6.5.4 introduce new concurrency limits?



We upgraded to 6.5.4 this weekend, and I'm now noticing lots of queued jobs, saying that "The maximum number of concurrent historical searches for this user based on their role quota has been reached. concurrency_limit=3". We did not see these messages prior to this release (6.4.1). Is this an new and/or updated config value?

Ultra Champion

You still rocking Search Head Pooling (deprecated)? Are your Search Heads VMs with low specs?

If I recall correctly, when Search Head Clustering first came out, a common complaint was that the total concurrent search capacity was still being calculated on a per Search Head basis, which wasn't really effective given how Search Head Clustering better pools and manages the collective Search Head's resources. Don't forget that a Splunk Instances concurrency is limited by its hardware.

As a result, I think there was a tweak to make the concurrency work a little different on a Search Head Cluster. It may have even come out back in 6.3.x. I'm having trouble finding corresponding docs to validate this, but here's what I did find that might still be effective for you:

While I recognize you might be using Search Head Pooling, it might still be worth investigating if some of the Search Head Clustering Concurrency settings are trying to kick in on the Search Heads. Perhaps the Search Heads just assume they are Clustered cause why would anyone still be Pooling 😉 (I had to get a dig in there for tough love).

Revered Legend

Can you run this search and see if the concurrent searches have increased since last week?

index=_internal sourcetype=splunkd source=*metrics.log group=searech_concurrency 


index=_internal sourcetype=splunkd source=*metrics.log group=searech_concurrency | timechart max(active_*) as max_concurrent_*
0 Karma


This is strange -

index=_internal sourcetype=scheduler maximum number of concurrent historical |  timechart max(concurrency_limit)  

I see the max number drop in half after upgrading from 6.5.2 to 6.5.4. (From 51 to 27)

max_searches_per_cpu = 2
base_max_searches = 6
CPUs: 24 (12 hyper threaded)

So as I understand this number should be 54

From limits.conf:
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches

Looks like something has changed in 6.5.4

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!