The reason for memory growth is auto tuning for max_inactive and lowater_inactive configurations in limits.conf. With auto tuning, max_inactive = 96 ( if total system memory is < 8GB) max_inactive = 1024 ( if total system memory is >= 8GB and < 26GB ) max_inactive = 32768 ( if total system memory is >= 26GB) lowater_inactive = (max_inactive/3) max_inactive = <integer>
* The Maximum number of inactive input channel configurations to keep in cache.
* Each source/sourcetype/host combination requires an independent input
channel, which contains all relevant settings for ingestion.
* When set to 'auto', the Splunk platform will tune this setting based on the
physical RAM present in the server at startup.
* Increasing this number might help with low ingestion throughput when there
are no blocked queues (i.e., no 'blocked=true' events for 'group=queue' in
metrics.log), and splunkd is creating a very high number of new input
channels (see the value of 'new_channels' in
'group=map, name=pipelineinputchannel', also in metrics.log), usually in the
order of thousands. However, this action is only effective when those input
channels could have been reused: for example, the source, sourcetype, and
host fields are not generated randomly and tend to be reused within the
lifetime of cached channel entries.
* Default: auto
lowater_inactive = <integer>
* Size of the inactive input channel cache after which entries will be
considered for recycling: having its memory reused for storing settings
for a different input channel.
* When set to 'auto', the Splunk platform will tune this setting value based
on the value of 'max_inactive'.
* Default: auto As a result Universal forwarder/Search Head is creating a minimum cache of inactive channels as per lowater_inactive configuration. However these high settings are useful for only Indexer and Heavy forwarder. For edge Universal forwarder and search head these high values don't matter. Workaround: Set `max_inactive` as low as possible. Example [input_channels] max_inactive=10
... View more
