Why do I receive error "system wide historical concurrent searches quota has been reached"?

[ankithreddy777]

we have 10 indexers with 16 CPU cores each. Our replication is 4
base_searches=6 and max_searches_per_cpu =1.

I am getting an error stating
"system wide concurrent searches has be reached, maximum=38 current=38".

I assume that if indexer has 16 cores, system wide limit should = base_searches+max_searches per_cpu*number of cpu=6+1*16 =22

But why I am getting the error with 38 limit?

[sowings]

This limit is actually enforced by the search head, not the indexers. The limit of 38 would be reflected as 6 + the number of cores in your search head, which I'm guessing is 32.

[ankithreddy777]

My search head has 16 cores with max_searches_per_cpu setting =1

[somesoni2]

Are you using Search Head Clustering?

[ankithreddy777]

we are not using search head clustering

[somesoni2]

Can you run following search see what is the numberOfCores identified by Splunk?

| rest /services/server/info splunk_server=local | table host numberOfCores

[ankithreddy777]

Its giving number of cores on server(which is SH)= 16