You shouldn't need to do anything. If you do hit the max concurrent search limit, once a search completes, it frees up a spot for the next search in line. Eventually all searches will complete.
It is not necessarily bad if you keep encountering this error so long a the max concurrent searches is not reached. If you consistently reach max concurrent searches you may want to look into staggering your searches.
If you have hardware to support allocation of more CPU resources to search execution, then it is possible to increase the max number of concurrent searches.
The number of concurrent searches is based on CPU and affected by 2 settings in limits.conf. From limits.conf.spec:
> [search] > max_searches_per_cpu = <int> > * the maximum number of concurrent searches per CPU. The system-wide number of searches > * is computed as max_searches_per_cpu x number_of_cpus + 2 > * Defaults to 2 > [scheduler] > max_searches_perc = <integer> > * the maximum number of searches the scheduler can run, as a percentage > * of the maximum number of concurrent searches, see [search] max_searches_per_cpu > * for how to set the system wide maximum number of searches > * Defaults to 25
For an 8 CPU box, the default number of
max_searches_per_cpu is 18 (4x4cpu+2). It is likely, however, that such a server is capable of supporting more searches per CPU so it is safe to increment to
Additionally, if you are running many scheduled searches for alerts or dashboards, you can find a more equitable division with max_searches_perc than the default 25%.
These settings will allow you to maximize your hardware. If you find this is not adequate, consider adding a server. In general, the recommended approach to scaling is to add more CPUs via additional Splunk servers so the workload of search execution can be shared.
Note that the default was bumped to
max_searches_per_cpu=4 in 4.1 (or possibly earlier). So the "Defaults to 2" is no longer accurate. I had bumped this to "3" years ago in my
local/limits.conf, so after and upgrade I end up with a lower value than the default. Just a heads up!
maxsearchespercpu=4 could be a typo. Splunk supports maximum of 2 searches per CPU.
so it should be maxsearchespercpu=2 or less.