Installed Universal forwarder and no inputs are added yet, still gradual memory growth.
Why there is constant memory growth with Universal Forwarder?
More importantly in the K8 cluster setting, every extra MB memory usage matters.
The reason for memory growth is auto tuning for max_inactive and lowater_inactive configurations in limits.conf.
With auto tuning,
max_inactive = 96 ( if total system memory is < 8GB)
max_inactive = 1024 ( if total system memory is >= 8GB and < 26GB )
max_inactive = 32768 ( if total system memory is >= 26GB)
lowater_inactive = (max_inactive/3)
max_inactive = <integer> * The Maximum number of inactive input channel configurations to keep in cache. * Each source/sourcetype/host combination requires an independent input channel, which contains all relevant settings for ingestion. * When set to 'auto', the Splunk platform will tune this setting based on the physical RAM present in the server at startup. * Increasing this number might help with low ingestion throughput when there are no blocked queues (i.e., no 'blocked=true' events for 'group=queue' in metrics.log), and splunkd is creating a very high number of new input channels (see the value of 'new_channels' in 'group=map, name=pipelineinputchannel', also in metrics.log), usually in the order of thousands. However, this action is only effective when those input channels could have been reused: for example, the source, sourcetype, and host fields are not generated randomly and tend to be reused within the lifetime of cached channel entries. * Default: auto lowater_inactive = <integer> * Size of the inactive input channel cache after which entries will be considered for recycling: having its memory reused for storing settings for a different input channel. * When set to 'auto', the Splunk platform will tune this setting value based on the value of 'max_inactive'. * Default: auto
As a result Universal forwarder is creating a minimum cache of inactive channels as per lowater_inactive configuration.
However these high settings are useful for only Indexer and Heavy forwarder. For edge Universal forwarder these high values don't matter.
Workaround:
Set `max_inactive` as low as possible.
Example
[input_channels]
max_inactive=10
Unless you deliberately disable inputs there are at least some internal splunk inputs enabled by default right after the installation. So splunk reads its own logs and wants to send them to the indexers, as configured in outputs.conf. You can't therefore say that you just installed the UF and didn't enable any inputs.
The reason for memory growth is auto tuning for max_inactive and lowater_inactive configurations in limits.conf.
With auto tuning,
max_inactive = 96 ( if total system memory is < 8GB)
max_inactive = 1024 ( if total system memory is >= 8GB and < 26GB )
max_inactive = 32768 ( if total system memory is >= 26GB)
lowater_inactive = (max_inactive/3)
max_inactive = <integer> * The Maximum number of inactive input channel configurations to keep in cache. * Each source/sourcetype/host combination requires an independent input channel, which contains all relevant settings for ingestion. * When set to 'auto', the Splunk platform will tune this setting based on the physical RAM present in the server at startup. * Increasing this number might help with low ingestion throughput when there are no blocked queues (i.e., no 'blocked=true' events for 'group=queue' in metrics.log), and splunkd is creating a very high number of new input channels (see the value of 'new_channels' in 'group=map, name=pipelineinputchannel', also in metrics.log), usually in the order of thousands. However, this action is only effective when those input channels could have been reused: for example, the source, sourcetype, and host fields are not generated randomly and tend to be reused within the lifetime of cached channel entries. * Default: auto lowater_inactive = <integer> * Size of the inactive input channel cache after which entries will be considered for recycling: having its memory reused for storing settings for a different input channel. * When set to 'auto', the Splunk platform will tune this setting value based on the value of 'max_inactive'. * Default: auto
As a result Universal forwarder is creating a minimum cache of inactive channels as per lowater_inactive configuration.
However these high settings are useful for only Indexer and Heavy forwarder. For edge Universal forwarder these high values don't matter.
Workaround:
Set `max_inactive` as low as possible.
Example
[input_channels]
max_inactive=10