Why is there Constant Memory growth with Universal Forwarder?

[hrawat_splunk]

Installed Universal forwarder and no inputs are added yet, still gradual memory growth.
Why there is constant memory growth with Universal Forwarder?
More importantly in the K8 cluster setting, every extra MB memory usage matters.

[PickleRick]

Unless you deliberately disable inputs there are at least some internal splunk inputs enabled by default right after the installation. So splunk reads its own logs and wants to send them to the indexers, as configured in outputs.conf. You can't therefore say that you just installed the UF and didn't enable any inputs.

[hrawat_splunk · Accepted solution]

The reason for memory growth is auto tuning for max_inactive and lowater_inactive configurations in limits.conf.
With auto tuning, 
max_inactive = 96 ( if total system memory is < 8GB)
max_inactive = 1024  ( if total system memory is >= 8GB and < 26GB )
max_inactive = 32768  ( if total system memory is >= 26GB)
lowater_inactive = (max_inactive/3)

max_inactive = <integer>
* The Maximum number of inactive input channel configurations to keep in cache.
* Each source/sourcetype/host combination requires an independent input
  channel, which contains all relevant settings for ingestion.
* When set to 'auto', the Splunk platform will tune this setting based on the
  physical RAM present in the server at startup.
* Increasing this number might help with low ingestion throughput when there
  are no blocked queues (i.e., no 'blocked=true' events for 'group=queue' in
  metrics.log), and splunkd is creating a very high number of new input
  channels (see the value of 'new_channels' in
  'group=map, name=pipelineinputchannel', also in metrics.log), usually in the
  order of thousands. However, this action is only effective when those input
  channels could have been reused: for example, the source, sourcetype, and
  host fields are not generated randomly and tend to be reused within the
  lifetime of cached channel entries.
* Default: auto

lowater_inactive = <integer>
* Size of the inactive input channel cache after which entries will be
  considered for recycling: having its memory reused for storing settings
  for a different input channel.
* When set to 'auto', the Splunk platform will tune this setting value based
  on the value of 'max_inactive'.
* Default: auto

As a result Universal forwarder is creating a minimum cache of inactive channels as per lowater_inactive configuration.
However these high settings are useful for only Indexer and Heavy forwarder. For edge Universal forwarder these high values don't matter.

Workaround:
Set `max_inactive` as low as possible.

Example
[input_channels]
max_inactive=10