streamfwd high memory usage on Linux

[YuriSpirin]

Hello!

I'm setting up Stream Forwarders on Linux DNS servers to index DNS traffic. Splunk_TA_Stream is deployed to Universal Forwarders by Splunk Deployment Server with all relevant configs and is working correctly except one thing: streamfwd process consumes 800-1000 MB of RAM which I think is a huge overhead for DNS servers with 1,5-2 GB RAM. Configuring queue sizes, bandwidth limits and so on in streamfwd.conf makes no difference. I see such behavior both on heavy loaded DNS servers (stream:dns traffic around 8-10 MBps) and on idle test servers - memory usage by streamfwd is +- the same.

So I have 2 questions:

1. Is there a way to limit streamfwd memory usage somehow by config settings?

2. How is streamfwd launched? I mean where is the line of code in Stream scripts or elsewhere by which /opt/splunkforwarder/etc/apps/Splunk_TA_Stream/bin/linux_x86_x64/streamfwd executable is run? I think of experiment by prepending this command line with ulimit or similar Linux command to set memory restrictions.

Or, maybe, such memory consumption is not a bug or misconfiguration but is by design?

Splunk Enterprise 7.2.6, Splunk_TA_Stream 7.2.0, Oracle Linux Server release 7.8.

[cameronjust]

Hi,

Did you ever get a solution to this? I'm seeing the same on Windows as well.

With no streams configured the streamfwd.exe process uses 5MB. The second I turn on just one stream all agents start hogging 700MB of memory.

This is kind of a deal breaker for a host with only 4GB of RAM. Infrastructure people really didn't like that and are reluctant to install site wide.

It's not isolated to busy hosts either. Quiet dev/test hosts with minimal traffic exhibited the same behaviours.

I've been playing around with enabling/disabling various streams to see if I can pinpoint the main culprit but no luck yet.

Thanks