Splunk Enterprise

Did 6.5.4 introduce new concurrency limits?

a212830
Champion

Hi,

We upgraded to 6.5.4 this weekend, and I'm now noticing lots of queued jobs, saying that "The maximum number of concurrent historical searches for this user based on their role quota has been reached. concurrency_limit=3". We did not see these messages prior to this release (6.4.1). Is this an new and/or updated config value?

sloshburch
Splunk Employee
Splunk Employee

You still rocking Search Head Pooling (deprecated)? Are your Search Heads VMs with low specs?

If I recall correctly, when Search Head Clustering first came out, a common complaint was that the total concurrent search capacity was still being calculated on a per Search Head basis, which wasn't really effective given how Search Head Clustering better pools and manages the collective Search Head's resources. Don't forget that a Splunk Instances concurrency is limited by its hardware.

As a result, I think there was a tweak to make the concurrency work a little different on a Search Head Cluster. It may have even come out back in 6.3.x. I'm having trouble finding corresponding docs to validate this, but here's what I did find that might still be effective for you:

While I recognize you might be using Search Head Pooling, it might still be worth investigating if some of the Search Head Clustering Concurrency settings are trying to kick in on the Search Heads. Perhaps the Search Heads just assume they are Clustered cause why would anyone still be Pooling 😉 (I had to get a dig in there for tough love).

somesoni2
SplunkTrust
SplunkTrust

Can you run this search and see if the concurrent searches have increased since last week?

index=_internal sourcetype=splunkd source=*metrics.log group=searech_concurrency 

OR

index=_internal sourcetype=splunkd source=*metrics.log group=searech_concurrency | timechart max(active_*) as max_concurrent_*
0 Karma

jplumsdaine22
Influencer

This is strange -

index=_internal sourcetype=scheduler maximum number of concurrent historical |  timechart max(concurrency_limit)  

I see the max number drop in half after upgrading from 6.5.2 to 6.5.4. (From 51 to 27)

max_searches_per_cpu = 2
base_max_searches = 6
CPUs: 24 (12 hyper threaded)

So as I understand this number should be 54

From limits.conf:
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches

Looks like something has changed in 6.5.4

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...