We upgraded to 6.5.4 this weekend, and I'm now noticing lots of queued jobs, saying that "The maximum number of concurrent historical searches for this user based on their role quota has been reached. concurrency_limit=3". We did not see these messages prior to this release (6.4.1). Is this an new and/or updated config value?
You still rocking Search Head Pooling (deprecated)? Are your Search Heads VMs with low specs?
If I recall correctly, when Search Head Clustering first came out, a common complaint was that the total concurrent search capacity was still being calculated on a per Search Head basis, which wasn't really effective given how Search Head Clustering better pools and manages the collective Search Head's resources. Don't forget that a Splunk Instances concurrency is limited by its hardware.
As a result, I think there was a tweak to make the concurrency work a little different on a Search Head Cluster. It may have even come out back in 6.3.x. I'm having trouble finding corresponding docs to validate this, but here's what I did find that might still be effective for you:
Welcome to Splunk Enterprise 6.3 Release Notes -> What's New in 6.3 -> Search Head Clustering Improvements. Performance optimization, scalability, and management improvements. Support for Windows OS.
While I recognize you might be using Search Head Pooling, it might still be worth investigating if some of the Search Head Clustering Concurrency settings are trying to kick in on the Search Heads. Perhaps the Search Heads just assume they are Clustered cause why would anyone still be Pooling 😉 (I had to get a dig in there for tough love).