topic Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs in Splunk Search

Splunk maxing out at 11 realtime searches despit having 16 CPUs

JeremyHagan — Tue, 29 Sep 2020 11:07:00 GMT

Hi,

I have a single-server instance of Splunk with 16 cores. According to my research the maximum number of realtime searches should be:
max real-time searches = max_rt_search_multiplier x max historical searches
and
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches

So substituting in values from limits.conf I see:
max real-time searches = 1 x (1 x 16 + 6) = 22

But in my scheduler.log I get:
The maximum number of concurrent scheduled searches has been reached (limits: historical=11, realtime=11). historical=0, realtime=12 ready-to-run scheduled searches are pending.

Suspiciously these two numbers add up to 22. What am I missing here? I thought realtime should be 22.

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

inventsekar — Thu, 22 Sep 2016 03:30:31 GMT

in my scheduler.log I get:
The maximum number of concurrent scheduled searches has been reached (limits: historical=11, realtime=11). historical=0, realtime=12 ready-to-run scheduled searches are pending.

i think, the (limits: historical=11, realtime=11) was misleading.
it should have said - (limits: historical=11 OR realtime=11). we should not add 11+11=22. its either 11 historical or 11 real time searches.

update -

can you run this for last 24hrs or last 7 days and update us few results...

index=_internal sourcetype=splunkd source=*metrics.log group=search_concurrency "system total" | table active_hist_searches active_realtime_searches

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

JeremyHagan — Thu, 22 Sep 2016 03:34:38 GMT

But why is it 11? According to the doco it should be 22.

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

inventsekar — Thu, 22 Sep 2016 03:55:44 GMT

may we know your limits.conf [search] configuration please.
also is this on search head cluster? ur splunk version also, please

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

JeremyHagan — Tue, 29 Sep 2020 11:07:03 GMT

From running splunk.exe show config limits I get the following for the [search] stanza. To my knowledge these are all defaults:
[search]
allow_batch_mode=true
allow_inexact_metasearch=false
base_max_searches=6
batch_retry_max_interval=300
batch_retry_min_interval=5
batch_retry_scaling=1.5
batch_search_max_index_values=10000000
batch_wait_after_end=900
cache_ttl=300
chunk_multiplier=5
default_allow_queue=true
default_save_ttl=604800
dispatch_dir_warning_size=2000
dispatch_quota_retry=4
dispatch_quota_sleep_ms=100
enable_history=true
failed_job_ttl=86400
fetch_remote_search_log=disabledSavedSearches
fieldstats_update_freq=0
fieldstats_update_maxperiod=60
load_remote_bundles=false
long_search_threshold=2
max_chunk_queue_size=1000000
max_combiner_memevents=50000
max_count=500000
max_history_length=1000
max_id_length=150
max_macro_depth=100
max_rawsize_perchunk=100000000
max_results_perchunk=2500
max_rt_search_multiplier=1
max_searches_per_cpu=1
max_tolerable_skew=60
max_workers_searchparser=5
min_freq=0.01
min_prefix_len=1
min_results_perchunk=100
multi_threaded_setup=0
preview_duty_cycle=0.25
queued_job_check_freq=1
realtime_buffer=10000
reduce_duty_cycle=0.25
reduce_freq=10
remote_timeline=true
remote_timeline_connection_timeout=5
remote_timeline_fetchall=1
remote_timeline_min_peers=1
remote_timeline_receive_timeout=10
remote_timeline_send_timeout=10
remote_timeline_touchperiod=300
remote_ttl=600
replication_file_ttl=600
replication_period_sec=60
result_queue_max_size=100000000
results_queue_min_size=10
rr_max_sleep_ms=1000
rr_min_sleep_ms=10
rr_sleep_factor=2
search_process_mode=auto
stack_size=4194304
status_buckets=0
status_cache_size=10000
summary_mode=all
sync_bundle_replication=auto
target_time_perchunk=2000
track_indextime_range=true
truncate_report=false
ttl=600
write_multifile_results_out=true

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

JeremyHagan — Thu, 22 Sep 2016 04:01:50 GMT

And it is a single-server instance, so indexer and search head all on one server. Running Splunk 6.2.6

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

inventsekar — Tue, 29 Sep 2020 11:07:05 GMT

can you run this for last 24hrs or last 7 days and update us few results...
index=_internal sourcetype=splunkd source=*metrics.log group=search_concurrency "system total" | table active_hist_searches active_realtime_searches

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

JeremyHagan — Thu, 22 Sep 2016 04:50:06 GMT

Even for the last 4 hours that returns 240,000+ results and they seem to read 0,0.

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

JeremyHagan — Tue, 29 Sep 2020 11:07:13 GMT

If I sort by -active_realtime_searches then I get 13 as the max number of realtime searches in the last 24 hour period and 15 for historical.

I do get a row of 15 historical and 11 reatime

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

inventsekar — Thu, 22 Sep 2016 05:09:16 GMT

15 and 11 on the same line ah?!?
totally 26 searches ah?!?
One more question - how you found out the CPU cores please.. are you sure about the number of CPU cores on this system?!?

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

JeremyHagan — Thu, 22 Sep 2016 05:34:24 GMT

I'm 100% sure about the number of cores. Checked through Task Manager on Windows, plus I administer the VM it is running on and it has 2 x sockets with 8 cores each presented to it.

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

inventsekar — Thu, 22 Sep 2016 05:42:20 GMT

Oh Ok, great..
May I know, 15 and 11 on the same line ah?!?
totally 26 searches ah?!

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

JeremyHagan — Thu, 22 Sep 2016 05:55:43 GMT

Correct. On the same line.

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

inventsekar — Thu, 22 Sep 2016 05:58:52 GMT

Means, its saying only 11 and 11 searches together.. but, above one says 26 searches?!?

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

Masa — Thu, 22 Sep 2016 06:15:06 GMT

I believe it is a simple reason. scheduled searches is 50% of total historical search or realtime search. In your case, hist search max is 22. So, scheduled search is 50% of 22 = 11. Same for real-time search. 11 is max for real-time scheduled searches. It is in limits.conf.spec file.

Old info in wiki.splunk.com still works in general.
http://wiki.splunk.com/Community:TroubleshootingSearchQuotas

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

inventsekar — Thu, 22 Sep 2016 06:20:37 GMT

Great, though I didn't solve this, I am glad, we troubleshooted this.. maybe, an upvote, to cheer me;)

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

JeremyHagan — Tue, 29 Sep 2020 11:07:19 GMT

Sound plausible. From the [scheduler] section:
max_searches_perc: defaults to 50.

I ended up changing the rt search multiplier from 1 to 2. I can't find any good data on what value it is safe to set this to. Some people have said 4 and one person set it to 8. Setting it to 2 got me out of trouble in this case.

I wish the documentation was clearer on this.

Re: Splunk maxing out at 11 realtime searches despit having 16 CPUs

Masa — Thu, 22 Sep 2016 06:28:10 GMT

Once you changed the default value, you are easier to reach high resource usage and performance issue. It all depends on how your searches finishes quickly and avoid long running concurrent searches. I had a user standalone and set to 4 and once enabling Report acceleration (a lot), their end users complained performance issue. They changed to 2. Still not enough to avoid performance issue. They had to disable Report acceleration. Again, basically, changing the default value will push system resource work harder. How much hard they can work without bad user experience? That all depends.