The limits.conf parameter fetch_remote_search_log can contribute to search performance problems for sizable deployments, especially those with a high number of searches and use of summary indexing.
This parameter is enabled by default in 5.x versions; however, btool does not produce a result because there is not a default/limits.conf entry in several of the 5.x versions. There is however an entry in the Splunk spec file in etc/system/README/limits.conf.spec
limits.conf.spec
fetch_remote_search_log =
* If true, will attempt to fetch the search.log from every search peer at the end of the search and store in the job dispatch dir on the search head.
* Defaults to true
This setting is intended to make search logs available to the User on a search head without having to go directly to a search peer and retrieving the search artifact’s search.log.
This functionality is not needed for scheduled searches.
Included are several bugs to disable this setting by default, and for more control over the functionality in terms of enabling it for specific types of searches, and not all.
SPL-76948: disable search.log collection
SPL-76948: Allow a more fine-grained control of remote search.log collection
Below are a few searches to help identify if this setting is a possible factor:
Compare before and after:
index=_internal SavedSplunker | timechart avg(run_time) median(run_time)
index=_internal source=scheduler.log | timechart span=1h count, sum(eval(run_time/60)) as run_time
The following will graph a growing latency.
index=_internal SavedSplunker | eval lag=dispatch_time - scheduled_time | timechart avg(lag)
To address these symptoms the following configuration should be added to search heads.
Eg:
$SPLUNK_HOME/etc/system/local/limits.conf
[search]
fetch_remote_search_log = false
