Splunk Search

How to increase the maximum number of real-time searches

Path Finder

I am trying to do a stress test on a new server in a fresh Splunk environment. I would like to increase the number of real-time searches allowed and see how much of a performance hit the server takes. I know I need to edit limits.conf, but I am not certain what stanza to add and what values to use.

Anyone with experience in tweaking limits.conf is what I am looking for. Is it best to start off by just adding a search stanza with: max_rt_search_multiplier = 2

or is it better to add:

max_searches_per_cpu = 2 ?

Our goal to is see how many concurrent real-time searches we can run before we start having a substantial performance hit.

0 Karma

Splunk Employee
Splunk Employee

changing the max_rt_search_multiplier is the way to go. changing max_searches_per_cpu and base_max_searches will change also affect the real-time limit, but will alter the limit for historical searches too.

max real-time searches = max_rt_search_multiplier x (max_searches_per_cpu * + base_max_searches)

And to address woodcock's comment, there is a setting that you can tweak which will allow you to optionally trade performance for latency. It will run real-time searches with higher latency but generally use far less system resources. The setting is under

[realtime]
indexed_realtime_use_by_default = true/false (defaults to false. set to true for less resource usage but higher latency)

Esteemed Legend

I can tell you that answer: ONE! Unless you have designated your entire cluster to the purpose of running Real-Time Searches, don't run any.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!