Splunk Search

How to configure limits.conf to make the most of our hardware and improve search performance?

dwh_splunk
Explorer

My company has two massive machines as search heads: 256GB RAM and 24 cores each.
The indexers are equipped just fine as well.

Then we have searches that query a lot of data.
We notice:
- The RAM is never used, at most 50-70GB
- The search times grow exponentially. Timerange 1 hour -> fine. Timerange 8h -> 20x duration

What could be the limiting factors that hinder performance?
What values in limits.conf promise better performance by tweaking?

The settings in limits.conf are described shortly in http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf
but don't guide me to a better understanding.
I.e what are good values for max_mem_usage_mb and maxresultrows?

Any help and hints appreciated

splunkLPN
Path Finder

Just to know, did you modify one of this settings ?

"search_process_memory_usage_threshold =
* To be active, this setting requires setting: enable_memory_tracker = true
* Signifies the maximum memory in MB the search process can consume in RAM.
* Search processes violating the threshold will be terminated.
* If the value is set to zero, then splunk search processes are allowed
to grow unbounded in terms of in memory usage.

* The default value is set to 4000MB or 4GB.

stack_size =
* The stack size (in bytes) of the thread executing the search.
* Defaults to 4194304 (4 MB)

status_cache_size =
* The number of search job status data splunkd can cache in RAM. This cache
improves performance of the jobs endpoint
* Defaults to 10000"

What is you usage per core ?

perhaps did you already made empirical settings and compare the processing time obtained for the same search ?

masonmorales
Influencer

The indexers do the majority of the work, and Splunk scales horizontally. What does your indexing infrastructure look like? Massive search heads give you massive concurrent search capability, but not faster search performance. For that, you need to add more/faster indexers.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...