Join the Conversation
- Find Answers
- :
- Splunk Administration
- :
- Deployment Architecture
- :
- Re: splunkforwarder out of memory
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
splunkforwarder out of memory
I've been experimenting some out of memory issues in my server lately, basically the oom-killer
is called and one or more processes are killed. Among the processes that get killed there's always splunkforwarder.
After some testing, I decided to remove splunkforwarder from my server's boot and all problems stopped. If at any point in time I start the process, I get a new oom-killer issue.
Server is a small instance in amazon's ec2, using Ubuntu 12.04 LTS. This are my deploy commands:
/opt/splunkforwarder/bin/splunk start --accept-license
/opt/splunkforwarder/bin/splunk install app ... -auth admin:changeme
/opt/splunkforwarder/bin/splunk login -auth admin:changeme
/opt/splunkforwarder/bin/splunk edit user admin -password df5...f13
/opt/splunkforwarder/bin/splunk list forward-server
/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log
The questions are:
- Do you guys know about any memory leaks, or memory usage issues in splunkforwarder?
- Any idea on how to reduce the memory usage? Any configuration parameter I can modify? I don't care about having all the information sent immediately to splunkstorm; if there is a compromise between speed and memory use, I would choose low memory use and slow speed.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This issue is generally when you have generalized a path to monitor.
Example [monitor:///var/logs/*]
This causes the issue. I had faced the same when we had added a generalized path (for oracle logs) the splunkd process was taking up heavy memory.
After changing path to specific log file to be monitored the memory usage settled down.
Example : [monitor:///var/log/messages]
Give it a go, hope it helps.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Down voted since I'm only monitoring one file:
/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log
The good thing is that your comment lets me know that this is a splunk bug. It shouldn't take more memory to monitor a large number of files.
@Splunk developers: Please fix.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Not that I'm aware of.
Have you looked at the 5.0.2 version. It's been out for some time now.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Any ideas on what this could be about? Is this a known issue? When should I expect a fix? Anything I can do to help with testing the fix?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
dpkg -i splunkforwarder-5.0.1-143156-linux-2.6-amd64.deb
Amazon ec2 AMI: ubuntu/images/ebs/ubuntu-precise-12.04-amd64-server-20121001
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
what version are you running?
Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!
Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.
[Puzzles] Solve, Learn, Repeat: Matching cron expressions
Design, Compete, Win: Submit Your Best Splunk Dashboards for a .conf26 Pass
May 2026 Splunk Expert Sessions: Security & Observability
