- Splunk Answers
- :
- Splunk Administration
- :
- Deployment Architecture
- :
- Re: splunkforwarder out of memory
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
splunkforwarder out of memory
I've been experimenting some out of memory issues in my server lately, basically the oom-killer
is called and one or more processes are killed. Among the processes that get killed there's always splunkforwarder.
After some testing, I decided to remove splunkforwarder from my server's boot and all problems stopped. If at any point in time I start the process, I get a new oom-killer issue.
Server is a small instance in amazon's ec2, using Ubuntu 12.04 LTS. This are my deploy commands:
/opt/splunkforwarder/bin/splunk start --accept-license
/opt/splunkforwarder/bin/splunk install app ... -auth admin:changeme
/opt/splunkforwarder/bin/splunk login -auth admin:changeme
/opt/splunkforwarder/bin/splunk edit user admin -password df5...f13
/opt/splunkforwarder/bin/splunk list forward-server
/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log
The questions are:
- Do you guys know about any memory leaks, or memory usage issues in splunkforwarder?
- Any idea on how to reduce the memory usage? Any configuration parameter I can modify? I don't care about having all the information sent immediately to splunkstorm; if there is a compromise between speed and memory use, I would choose low memory use and slow speed.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This issue is generally when you have generalized a path to monitor.
Example [monitor:///var/logs/*]
This causes the issue. I had faced the same when we had added a generalized path (for oracle logs) the splunkd process was taking up heavy memory.
After changing path to specific log file to be monitored the memory usage settled down.
Example : [monitor:///var/log/messages]
Give it a go, hope it helps.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Down voted since I'm only monitoring one file:
/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log
The good thing is that your comment lets me know that this is a splunk bug. It shouldn't take more memory to monitor a large number of files.
@Splunk developers: Please fix.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Not that I'm aware of.
Have you looked at the 5.0.2 version. It's been out for some time now.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Any ideas on what this could be about? Is this a known issue? When should I expect a fix? Anything I can do to help with testing the fix?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
dpkg -i splunkforwarder-5.0.1-143156-linux-2.6-amd64.deb
Amazon ec2 AMI: ubuntu/images/ebs/ubuntu-precise-12.04-amd64-server-20121001
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
what version are you running?
Routing logs with Splunk OTel Collector for Kubernetes
Welcome to the Splunk Community!
Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM
