Deployment Architecture

splunkforwarder out of memory

andresriancho
Explorer

I've been experimenting some out of memory issues in my server lately, basically the oom-killer
is called and one or more processes are killed. Among the processes that get killed there's always splunkforwarder.

After some testing, I decided to remove splunkforwarder from my server's boot and all problems stopped. If at any point in time I start the process, I get a new oom-killer issue.

Server is a small instance in amazon's ec2, using Ubuntu 12.04 LTS. This are my deploy commands:

/opt/splunkforwarder/bin/splunk start --accept-license
/opt/splunkforwarder/bin/splunk install app ... -auth admin:changeme
/opt/splunkforwarder/bin/splunk login -auth admin:changeme
/opt/splunkforwarder/bin/splunk edit user admin -password df5...f13
/opt/splunkforwarder/bin/splunk list forward-server
/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log

The questions are:

  • Do you guys know about any memory leaks, or memory usage issues in splunkforwarder?
  • Any idea on how to reduce the memory usage? Any configuration parameter I can modify? I don't care about having all the information sent immediately to splunkstorm; if there is a compromise between speed and memory use, I would choose low memory use and slow speed.
Tags (1)
0 Karma

sinclairmachado
Explorer

This issue is generally when you have generalized a path to monitor.
Example [monitor:///var/logs/*]

This causes the issue. I had faced the same when we had added a generalized path (for oracle logs) the splunkd process was taking up heavy memory.
After changing path to specific log file to be monitored the memory usage settled down.
Example : [monitor:///var/log/messages]

Give it a go, hope it helps.

andresriancho
Explorer

Down voted since I'm only monitoring one file:

/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log

The good thing is that your comment lets me know that this is a splunk bug. It shouldn't take more memory to monitor a large number of files.

@Splunk developers: Please fix.

0 Karma

kristian_kolb
Ultra Champion

Not that I'm aware of.

Have you looked at the 5.0.2 version. It's been out for some time now.

0 Karma

andresriancho
Explorer

Any ideas on what this could be about? Is this a known issue? When should I expect a fix? Anything I can do to help with testing the fix?

0 Karma

andresriancho
Explorer

dpkg -i splunkforwarder-5.0.1-143156-linux-2.6-amd64.deb

Amazon ec2 AMI: ubuntu/images/ebs/ubuntu-precise-12.04-amd64-server-20121001

0 Karma

piebob
Motivator

what version are you running?

0 Karma