Monitoring Splunk

Why is the memory usage on the SearchHead/Forwarder really high?

kmcg1
New Member

Hi, I have two Standalone Search Heads that also act as Heavy Forwarders running Splunk 6.5.1 on RHEL 6.9.

The past few days both instances have experienced potential memory leaks. Each morning, around the same time, both servers reach 100% memory usage and the splunkd process is killed by the kernel. Looking at the Historical Charts - Average Physical Memory Usage, it shows the climb up to 100% memory usage is linear on both servers, occurring gradually over several hours. Looking through various logs it is apparent the splunkd process is using up all of the memory, at which point the kernel kills the splunkd process.

There have been no significant changes in the environment the past few days, and I am unsure why this is behavior is developing now. Is there any way to further troubleshoot the specifics of what may be causing the increase memory usage on these two servers?

Thanks in advance for any help!

0 Karma
Get Updates on the Splunk Community!

Say goodbye to manually analyzing phishing and malware threats with Splunk Attack ...

In today’s evolving threat landscape, we understand you’re constantly bombarded with phishing and malware ...

AppDynamics is now part of Splunk Ideas

Hello Splunkers, We have exciting news for you! AppDynamics has been added to the Splunk Ideas Portal. Which ...

Advanced Splunk Data Management Strategies

Join us on Wednesday, May 14, 2025, at 11 AM PDT / 2 PM EDT for an exclusive Tech Talk that delves into ...