Tuning max searches on a summary indexing instance - how?

[the_wolverine]

I have an instance that I've set up to only run summary searches. Essentially, its a search head but no users connect directly to it and it only runs summary indexing searches.

I see a lot of the following errors in my splunkd.log:

WARN SavedSplunker - Maximum number (2) of concurrent scheduled searches reached. 16 ready-to-run scheduled searches pending.

Can I tune some parameters in limits.conf to better the performance? Right now, its telling me I'm maxing out at 2 concurrent searches and it should be able to handle more considering no users are connecting directly to it.

[the_wolverine · Accepted solution]

In general, we don't advise that you edit limits.conf unless you really know what you're doing.

In this situation, you should be able to modify the following settings in $SPLUNK_HOME/etc/system/local/limits.conf (listed below are the defaults in version 4.1):

[search]
max_searches_per_cpu = 4

[scheduler]
max_searches_perc = 25

Let's say your instance has 2 cpus. The number of (concurrent) searches per cpu, based on the default settings, will be 8 searches. For scheduled searches the default is 25% of that number so your max concurrent SCHEDULED searches (which applies to summary indexing searches) will be 2 concurrent searches.

If you're only running summary searches on this machine, you could raise the max_searches_perc up to 100 meaning that up to 8 scheduled searches can run concurrently.

If this system is not utilized by anything else, you could potentially raise the max_searches_per_cpu setting as well.

After modifying either of these settings make sure to monitor your system for a period of time to ensure it is not being overtaxed at any point in time.