topic Re: splunkforwarder out of memory in Deployment Architecture

splunkforwarder out of memory

andresriancho — Tue, 16 Apr 2013 21:21:26 GMT

I've been experimenting some out of memory issues in my server lately, basically the oom-killer
is called and one or more processes are killed. Among the processes that get killed there's always splunkforwarder.

After some testing, I decided to remove splunkforwarder from my server's boot and all problems stopped. If at any point in time I start the process, I get a new oom-killer issue.

Server is a small instance in amazon's ec2, using Ubuntu 12.04 LTS. This are my deploy commands:

/opt/splunkforwarder/bin/splunk start --accept-license
/opt/splunkforwarder/bin/splunk install app ... -auth admin:changeme
/opt/splunkforwarder/bin/splunk login -auth admin:changeme
/opt/splunkforwarder/bin/splunk edit user admin -password df5...f13
/opt/splunkforwarder/bin/splunk list forward-server
/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log

The questions are:

Do you guys know about any memory leaks, or memory usage issues in splunkforwarder?
Any idea on how to reduce the memory usage? Any configuration parameter I can modify? I don't care about having all the information sent immediately to splunkstorm; if there is a compromise between speed and memory use, I would choose low memory use and slow speed.

Re: splunkforwarder out of memory

piebob — Tue, 16 Apr 2013 21:29:01 GMT

what version are you running?

Re: splunkforwarder out of memory

andresriancho — Tue, 16 Apr 2013 21:35:54 GMT

dpkg -i splunkforwarder-5.0.1-143156-linux-2.6-amd64.deb

Amazon ec2 AMI: ubuntu/images/ebs/ubuntu-precise-12.04-amd64-server-20121001

Re: splunkforwarder out of memory

andresriancho — Tue, 23 Apr 2013 11:51:28 GMT

Any ideas on what this could be about? Is this a known issue? When should I expect a fix? Anything I can do to help with testing the fix?

Re: splunkforwarder out of memory

kristian_kolb — Tue, 23 Apr 2013 14:17:56 GMT

Not that I'm aware of.

Have you looked at the 5.0.2 version. It's been out for some time now.

Re: splunkforwarder out of memory

sinclairmachado — Tue, 23 Apr 2013 14:58:22 GMT

This issue is generally when you have generalized a path to monitor.
Example [monitor:///var/logs/*]

This causes the issue. I had faced the same when we had added a generalized path (for oracle logs) the splunkd process was taking up heavy memory.
After changing path to specific log file to be monitored the memory usage settled down.
Example : [monitor:///var/log/messages]

Give it a go, hope it helps.

Re: splunkforwarder out of memory

andresriancho — Tue, 23 Apr 2013 16:06:24 GMT

Down voted since I'm only monitoring one file:

/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log

The good thing is that your comment lets me know that this is a splunk bug. It shouldn't take more memory to monitor a large number of files.

@Splunk developers: Please fix.