Splunk Search

How to configure limits.conf to make the most of our hardware and improve search performance?


My company has two massive machines as search heads: 256GB RAM and 24 cores each.
The indexers are equipped just fine as well.

Then we have searches that query a lot of data.
We notice:
- The RAM is never used, at most 50-70GB
- The search times grow exponentially. Timerange 1 hour -> fine. Timerange 8h -> 20x duration

What could be the limiting factors that hinder performance?
What values in limits.conf promise better performance by tweaking?

The settings in limits.conf are described shortly in http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf
but don't guide me to a better understanding.
I.e what are good values for max_mem_usage_mb and maxresultrows?

Any help and hints appreciated

Path Finder

Just to know, did you modify one of this settings ?

"search_process_memory_usage_threshold =
* To be active, this setting requires setting: enable_memory_tracker = true
* Signifies the maximum memory in MB the search process can consume in RAM.
* Search processes violating the threshold will be terminated.
* If the value is set to zero, then splunk search processes are allowed
to grow unbounded in terms of in memory usage.

* The default value is set to 4000MB or 4GB.

stack_size =
* The stack size (in bytes) of the thread executing the search.
* Defaults to 4194304 (4 MB)

status_cache_size =
* The number of search job status data splunkd can cache in RAM. This cache
improves performance of the jobs endpoint
* Defaults to 10000"

What is you usage per core ?

perhaps did you already made empirical settings and compare the processing time obtained for the same search ?


The indexers do the majority of the work, and Splunk scales horizontally. What does your indexing infrastructure look like? Massive search heads give you massive concurrent search capability, but not faster search performance. For that, you need to add more/faster indexers.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...