I've been experimenting some out of memory issues in my server lately, basically the oom-killer
is called and one or more processes are killed. Among the processes that get killed there's always splunkforwarder.
After some testing, I decided to remove splunkforwarder from my server's boot and all problems stopped. If at any point in time I start the process, I get a new oom-killer issue.
Server is a small instance in amazon's ec2, using Ubuntu 12.04 LTS. This are my deploy commands:
/opt/splunkforwarder/bin/splunk start --accept-license /opt/splunkforwarder/bin/splunk install app ... -auth admin:changeme /opt/splunkforwarder/bin/splunk login -auth admin:changeme /opt/splunkforwarder/bin/splunk edit user admin -password df5...f13 /opt/splunkforwarder/bin/splunk list forward-server /opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log
The questions are:
This issue is generally when you have generalized a path to monitor.
This causes the issue. I had faced the same when we had added a generalized path (for oracle logs) the splunkd process was taking up heavy memory.
After changing path to specific log file to be monitored the memory usage settled down.
Example : [monitor:///var/log/messages]
Give it a go, hope it helps.
Down voted since I'm only monitoring one file:
/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log
The good thing is that your comment lets me know that this is a splunk bug. It shouldn't take more memory to monitor a large number of files.
@Splunk developers: Please fix.