After upgrade to 9.1.x and above overall increased system memory usage

[hrawat]

While there can be so many reasons for memory growth, one of the reason could be increased memory usage by idle search process pool(search-launcher).

index=_introspection component=PerProcess  host=<any one SH or IDX host>
| timechart span=5s sum(data.mem_used) as mem_usedMB by data.process_type useother=f usenull=f

Example

If memory usage by `search-launcher` is way higher than `search`

Then idle search process pool(search-launcher) is wasting system memory.

If you see above trend, we want to reduce idle search process pool.
There are several options to reduce idle search process pool in limits.conf

One option is to set enable_search_process_long_lifespan = false in server.conf( new option in 9.1 and above)

enable_search_process_long_lifespan = <boolean>
* Controls whether the search process can have a long lifespan.
* Configuring a long lifespan on a search process can optimize performance
  by reducing the number of new processes that are launched and old 
  processes that are reaped, and is a more efficient use of system 
  resources. 
* When set to "true": Splunk software does the following:
  * Suppresses increases in the configuration generation. See the 
    'conf_generation_include' setting for more information. 
  * Avoids unnecessary replication of search configuration bundles.
  * Allows a certain number of idle search processes to live.
  * Sets the size of the pool of search processes.
  * Checks memory usage before a search process is reused.
* When set to "false": The lifespan of a search process at the 50th 
  percentile is approximately 30 seconds. 
* NOTE: Do not change this setting unless instructed to do so by Splunk Support.
* Default: true

 
Why idle search process pool appears to be un-used(more idle searches compared to the actual number of searches running on peer)?

Before a search request is dispatched to peers, SHCs/SHs also need to first find  the common knowledge bundle across peers. On peer, only an idle search process created with matching common knowledge bundle is eligible for re-use. That's why in most cases idle search process pool remains un-used as overall idle search process pool is a collection of idle search processes  having association with different knowledge bundles. 

Now think of a scenario having multiple SHC clusters (example ES/ITSI/ad-hoc etc). Each SH cluster replicating it's own knowledge bundles. The idle search process pool is a collection of idle search processes  having association with different knowledge bundles from different search heads.

You can search enable_search_process_long_lifespan in limits.conf for the impact. It  controls lot of configs. 

But the main reason for memory growth is  max_search_process_pool (default 2048 idle search process pool).

max_search_process_pool = auto | <integer>
* The maximum number of search processes that can be launched to run searches
  in the pool of preforked search processes.
* The setting is valid if the 'enable_search_process_long_lifespan' setting  
  in the server.conf file is set to "true".
* Use this setting to limit the total number of running search processes on
  a search head or peer that is prevented from being overloaded or using
  high system resources (CPU, Memory, etc).
* When set to "auto": Splunk server determines the pool size by multiplying 
  the number of CPU cores and the allowed number of search processes (16). 
  The pool size is 64 at minimum.
* When set to "-1" or another negative value: The pool size is not limited.
* Has no effect on Windows or if "search_process_mode" is not "auto".
* Default: 2048

If an instance is running 1000 searches per minute, assuming bundle replication is not frequent, why to create 2048 idle searches pool when the max requirement is 1000?
With surplus memory resource, this is not an issue.
2048 idle searches pool is not ok for limited memory instances.