Knowledge Management

Tuning max searches on a summary indexing instance - how?

the_wolverine
Champion

I have an instance that I've set up to only run summary searches. Essentially, its a search head but no users connect directly to it and it only runs summary indexing searches.

I see a lot of the following errors in my splunkd.log:

WARN SavedSplunker - Maximum number (2) of concurrent scheduled searches reached. 16 ready-to-run scheduled searches pending.

Can I tune some parameters in limits.conf to better the performance? Right now, its telling me I'm maxing out at 2 concurrent searches and it should be able to handle more considering no users are connecting directly to it.

0 Karma
1 Solution

the_wolverine
Champion

In general, we don't advise that you edit limits.conf unless you really know what you're doing.

In this situation, you should be able to modify the following settings in $SPLUNK_HOME/etc/system/local/limits.conf (listed below are the defaults in version 4.1):

[search]
max_searches_per_cpu = 4

[scheduler]
max_searches_perc = 25

Let's say your instance has 2 cpus. The number of (concurrent) searches per cpu, based on the default settings, will be 8 searches. For scheduled searches the default is 25% of that number so your max concurrent SCHEDULED searches (which applies to summary indexing searches) will be 2 concurrent searches.

If you're only running summary searches on this machine, you could raise the max_searches_perc up to 100 meaning that up to 8 scheduled searches can run concurrently.

If this system is not utilized by anything else, you could potentially raise the max_searches_per_cpu setting as well.

After modifying either of these settings make sure to monitor your system for a period of time to ensure it is not being overtaxed at any point in time.

View solution in original post

the_wolverine
Champion

In general, we don't advise that you edit limits.conf unless you really know what you're doing.

In this situation, you should be able to modify the following settings in $SPLUNK_HOME/etc/system/local/limits.conf (listed below are the defaults in version 4.1):

[search]
max_searches_per_cpu = 4

[scheduler]
max_searches_perc = 25

Let's say your instance has 2 cpus. The number of (concurrent) searches per cpu, based on the default settings, will be 8 searches. For scheduled searches the default is 25% of that number so your max concurrent SCHEDULED searches (which applies to summary indexing searches) will be 2 concurrent searches.

If you're only running summary searches on this machine, you could raise the max_searches_perc up to 100 meaning that up to 8 scheduled searches can run concurrently.

If this system is not utilized by anything else, you could potentially raise the max_searches_per_cpu setting as well.

After modifying either of these settings make sure to monitor your system for a period of time to ensure it is not being overtaxed at any point in time.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...