Knowledge Management

Tuning max searches on a summary indexing instance - how?

the_wolverine
Champion

I have an instance that I've set up to only run summary searches. Essentially, its a search head but no users connect directly to it and it only runs summary indexing searches.

I see a lot of the following errors in my splunkd.log:

WARN SavedSplunker - Maximum number (2) of concurrent scheduled searches reached. 16 ready-to-run scheduled searches pending.

Can I tune some parameters in limits.conf to better the performance? Right now, its telling me I'm maxing out at 2 concurrent searches and it should be able to handle more considering no users are connecting directly to it.

0 Karma
1 Solution

the_wolverine
Champion

In general, we don't advise that you edit limits.conf unless you really know what you're doing.

In this situation, you should be able to modify the following settings in $SPLUNK_HOME/etc/system/local/limits.conf (listed below are the defaults in version 4.1):

[search]
max_searches_per_cpu = 4

[scheduler]
max_searches_perc = 25

Let's say your instance has 2 cpus. The number of (concurrent) searches per cpu, based on the default settings, will be 8 searches. For scheduled searches the default is 25% of that number so your max concurrent SCHEDULED searches (which applies to summary indexing searches) will be 2 concurrent searches.

If you're only running summary searches on this machine, you could raise the max_searches_perc up to 100 meaning that up to 8 scheduled searches can run concurrently.

If this system is not utilized by anything else, you could potentially raise the max_searches_per_cpu setting as well.

After modifying either of these settings make sure to monitor your system for a period of time to ensure it is not being overtaxed at any point in time.

View solution in original post

the_wolverine
Champion

In general, we don't advise that you edit limits.conf unless you really know what you're doing.

In this situation, you should be able to modify the following settings in $SPLUNK_HOME/etc/system/local/limits.conf (listed below are the defaults in version 4.1):

[search]
max_searches_per_cpu = 4

[scheduler]
max_searches_perc = 25

Let's say your instance has 2 cpus. The number of (concurrent) searches per cpu, based on the default settings, will be 8 searches. For scheduled searches the default is 25% of that number so your max concurrent SCHEDULED searches (which applies to summary indexing searches) will be 2 concurrent searches.

If you're only running summary searches on this machine, you could raise the max_searches_perc up to 100 meaning that up to 8 scheduled searches can run concurrently.

If this system is not utilized by anything else, you could potentially raise the max_searches_per_cpu setting as well.

After modifying either of these settings make sure to monitor your system for a period of time to ensure it is not being overtaxed at any point in time.

Get Updates on the Splunk Community!

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...