All Apps and Add-ons

streamfwd high memory usage on Linux

YuriSpirin
Explorer

Hello!

I'm setting up Stream Forwarders on Linux DNS servers to index DNS traffic. Splunk_TA_Stream is deployed to Universal Forwarders by Splunk Deployment Server with all relevant configs and is working correctly except one thing: streamfwd process consumes 800-1000 MB of RAM which I think is a huge overhead for DNS servers with 1,5-2 GB RAM. Configuring queue sizes, bandwidth limits and so on in streamfwd.conf makes no difference. I see such behavior both on heavy loaded DNS servers (stream:dns traffic around 8-10 MBps) and on idle test servers - memory usage by streamfwd is +- the same.

So I have 2 questions:

1. Is there a way to limit streamfwd memory usage somehow by config settings?

2. How is streamfwd launched? I mean where is the line of code in Stream scripts or elsewhere by which /opt/splunkforwarder/etc/apps/Splunk_TA_Stream/bin/linux_x86_x64/streamfwd executable is run? I think of experiment by prepending this command line with ulimit or similar Linux command to set memory restrictions.

Or, maybe, such memory consumption is not a bug or misconfiguration but is by design?

Splunk Enterprise 7.2.6, Splunk_TA_Stream 7.2.0, Oracle Linux Server release 7.8.

Labels (1)
Tags (3)
0 Karma

cameronjust
Path Finder

Hi,

Did you ever get a solution to this? I'm seeing the same on Windows as well.

With no streams configured the streamfwd.exe process uses 5MB. The second I turn on just one stream all agents start hogging 700MB of memory.

This is kind of a deal breaker for a host with only 4GB of RAM. Infrastructure people really didn't like that and are reluctant to install site wide.

It's not isolated to busy hosts either. Quiet dev/test hosts with minimal traffic exhibited the same behaviours.

I've been playing around with enabling/disabling various streams to see if I can pinpoint the main culprit but no luck yet.

 

Thanks

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...