How to increase the maximum number of concurrent historical searches?

[perlish]

Splunk warns that:

The system is approaching the maximum number of historical searches that can be run concurrently. current=55 maximum=68

...but the cpu and mem resouce of the system are free.

How to increase the maximum number of concurrent historical searches?

[Funderburg78]

Note: Splunk 7.1.x utilizes the following Defaults:
base_max_searches = 6
max_searches_per_cpu = 1

[bhawkins1]

Semi-unrelated note: If splunk is complaining about "maxsearches" - this isn't the same as the error described above. Instead, see documentation for map.

[hexx · Accepted solution]

If you want to fiddle with this limit, you'll have to set higher values than the default for base_max_searches or max_searches_per_cpu in $SPLUNK_HOME/etc/system/local/limits.conf. From limits.conf.spec:

[search]

base_max_searches = <int>
* A constant to add to the maximum number of searches, computed as a multiplier of the CPUs.
* Defaults to 4

max_searches_per_cpu = <int>
* The maximum number of concurrent historical searches per CPU. The system-wide limit of  historical searches is computed as: 
  max_hist_searches =  max_searches_per_cpu x number_of_cpus + base_max_searches
* Note: the maximum number of real-time searches is computed as: 
  max_rt_searches = max_rt_search_multiplier x max_hist_searches
* Defaults to 4

Do note, however, that 55 concurrent searches is already a pretty high number. If CPU is not a bottleneck (it seems that you have 16 cores on this machine), it's quite possible that I/O throughput and IOPS on the device hosting the indexes will be.

[ChrisG]

Note that in Splunk 5, the default value of max_searches_per_cpu is changed to 1 (from 4). See http://splunk-base.splunk.com/answers/70679/why-are-the-default-values-of-max_searches_per_cpu-and-b... for more information.