Splunk Search

How to configure limits.conf to make the most of our hardware and improve search performance?


My company has two massive machines as search heads: 256GB RAM and 24 cores each.
The indexers are equipped just fine as well.

Then we have searches that query a lot of data.
We notice:
- The RAM is never used, at most 50-70GB
- The search times grow exponentially. Timerange 1 hour -> fine. Timerange 8h -> 20x duration

What could be the limiting factors that hinder performance?
What values in limits.conf promise better performance by tweaking?

The settings in limits.conf are described shortly in http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf
but don't guide me to a better understanding.
I.e what are good values for max_mem_usage_mb and maxresultrows?

Any help and hints appreciated

Path Finder

Just to know, did you modify one of this settings ?

"search_process_memory_usage_threshold =
* To be active, this setting requires setting: enable_memory_tracker = true
* Signifies the maximum memory in MB the search process can consume in RAM.
* Search processes violating the threshold will be terminated.
* If the value is set to zero, then splunk search processes are allowed
to grow unbounded in terms of in memory usage.

* The default value is set to 4000MB or 4GB.

stack_size =
* The stack size (in bytes) of the thread executing the search.
* Defaults to 4194304 (4 MB)

status_cache_size =
* The number of search job status data splunkd can cache in RAM. This cache
improves performance of the jobs endpoint
* Defaults to 10000"

What is you usage per core ?

perhaps did you already made empirical settings and compare the processing time obtained for the same search ?


The indexers do the majority of the work, and Splunk scales horizontally. What does your indexing infrastructure look like? Massive search heads give you massive concurrent search capability, but not faster search performance. For that, you need to add more/faster indexers.

0 Karma
Get Updates on the Splunk Community!

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...