We upgraded to 6.5.4 this weekend, and I'm now noticing lots of queued jobs, saying that "The maximum number of concurrent historical searches for this user based on their role quota has been reached. concurrency_limit=3". We did not see these messages prior to this release (6.4.1). Is this an new and/or updated config value?
You still rocking Search Head Pooling (deprecated)? Are your Search Heads VMs with low specs?
If I recall correctly, when Search Head Clustering first came out, a common complaint was that the total concurrent search capacity was still being calculated on a per Search Head basis, which wasn't really effective given how Search Head Clustering better pools and manages the collective Search Head's resources. Don't forget that a Splunk Instances concurrency is limited by its hardware.
As a result, I think there was a tweak to make the concurrency work a little different on a Search Head Cluster. It may have even come out back in 6.3.x. I'm having trouble finding corresponding docs to validate this, but here's what I did find that might still be effective for you:
While I recognize you might be using Search Head Pooling, it might still be worth investigating if some of the Search Head Clustering Concurrency settings are trying to kick in on the Search Heads. Perhaps the Search Heads just assume they are Clustered cause why would anyone still be Pooling 😉 (I had to get a dig in there for tough love).
Can you run this search and see if the concurrent searches have increased since last week?
index=_internal sourcetype=splunkd source=*metrics.log group=searech_concurrency
index=_internal sourcetype=splunkd source=*metrics.log group=searech_concurrency | timechart max(active_*) as max_concurrent_*
This is strange -
index=_internal sourcetype=scheduler maximum number of concurrent historical | timechart max(concurrency_limit)
I see the max number drop in half after upgrading from 6.5.2 to 6.5.4. (From 51 to 27)
max_searches_per_cpu = 2
base_max_searches = 6
CPUs: 24 (12 hyper threaded)
So as I understand this number should be 54
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches
Looks like something has changed in 6.5.4