Knowledge Management

Tuning max searches on a summary indexing instance - how?

the_wolverine
Champion

I have an instance that I've set up to only run summary searches. Essentially, its a search head but no users connect directly to it and it only runs summary indexing searches.

I see a lot of the following errors in my splunkd.log:

WARN SavedSplunker - Maximum number (2) of concurrent scheduled searches reached. 16 ready-to-run scheduled searches pending.

Can I tune some parameters in limits.conf to better the performance? Right now, its telling me I'm maxing out at 2 concurrent searches and it should be able to handle more considering no users are connecting directly to it.

0 Karma
1 Solution

the_wolverine
Champion

In general, we don't advise that you edit limits.conf unless you really know what you're doing.

In this situation, you should be able to modify the following settings in $SPLUNK_HOME/etc/system/local/limits.conf (listed below are the defaults in version 4.1):

[search]
max_searches_per_cpu = 4

[scheduler]
max_searches_perc = 25

Let's say your instance has 2 cpus. The number of (concurrent) searches per cpu, based on the default settings, will be 8 searches. For scheduled searches the default is 25% of that number so your max concurrent SCHEDULED searches (which applies to summary indexing searches) will be 2 concurrent searches.

If you're only running summary searches on this machine, you could raise the max_searches_perc up to 100 meaning that up to 8 scheduled searches can run concurrently.

If this system is not utilized by anything else, you could potentially raise the max_searches_per_cpu setting as well.

After modifying either of these settings make sure to monitor your system for a period of time to ensure it is not being overtaxed at any point in time.

View solution in original post

the_wolverine
Champion

In general, we don't advise that you edit limits.conf unless you really know what you're doing.

In this situation, you should be able to modify the following settings in $SPLUNK_HOME/etc/system/local/limits.conf (listed below are the defaults in version 4.1):

[search]
max_searches_per_cpu = 4

[scheduler]
max_searches_perc = 25

Let's say your instance has 2 cpus. The number of (concurrent) searches per cpu, based on the default settings, will be 8 searches. For scheduled searches the default is 25% of that number so your max concurrent SCHEDULED searches (which applies to summary indexing searches) will be 2 concurrent searches.

If you're only running summary searches on this machine, you could raise the max_searches_perc up to 100 meaning that up to 8 scheduled searches can run concurrently.

If this system is not utilized by anything else, you could potentially raise the max_searches_per_cpu setting as well.

After modifying either of these settings make sure to monitor your system for a period of time to ensure it is not being overtaxed at any point in time.

Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...