Splunk Search

Why do I receive error "system wide historical concurrent searches quota has been reached"?

ankithreddy777
Contributor

we have 10 indexers with 16 CPU cores each. Our replication is 4
base_searches=6 and max_searches_per_cpu =1.

I am getting an error stating
"system wide concurrent searches has be reached, maximum=38 current=38".

I assume that if indexer has 16 cores, system wide limit should = base_searches+max_searches per_cpu*number of cpu=6+1*16 =22

But why I am getting the error with 38 limit?

0 Karma

sowings
Splunk Employee
Splunk Employee

This limit is actually enforced by the search head, not the indexers. The limit of 38 would be reflected as 6 + the number of cores in your search head, which I'm guessing is 32.

0 Karma

ankithreddy777
Contributor

My search head has 16 cores with max_searches_per_cpu setting =1

0 Karma

somesoni2
Revered Legend

Are you using Search Head Clustering?

0 Karma

ankithreddy777
Contributor

we are not using search head clustering

0 Karma

somesoni2
Revered Legend

Can you run following search see what is the numberOfCores identified by Splunk?

| rest /services/server/info splunk_server=local | table host numberOfCores
0 Karma

ankithreddy777
Contributor

Its giving number of cores on server(which is SH)= 16

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...