Monitoring Splunk

HIgh Physcial Memory Usage in Splunk searchhead VM

kartreddy4
New Member

Hi , I am currently facing this High Physical VM usage alert in splunk adhoc searchhead which is in VM environment (just did the HealthCheck). The specs are
Memory Used (MB) 17592186037778.105
Physical Memory Installed (MB) 15870.715
Memory used (%) 110846839841.7%

The THP aren't disabled and open files are 4096 against the recommended 8192. What could be the reason for this excessive memory usage and how to control it. Help would be highly appreciated.

Regards,
KJ

Tags (1)
0 Karma

deepashri_123
Motivator
0 Karma

tiagofbmm
Influencer

Well not having the THP and ulimits properly set could be the main issue

Check If this system is running Splunk as a user called "splunk," add the following to the bottom of the /etc/security/limits.conf file:

Should look like this your cat /opt/splunk/var/log/splunk/splunkd.log | grep ulimit

03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: virtual address space size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: data segment size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: resident memory size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: core file size: 0 bytes
03-01-2017 22:08:01.857 +0000 WARN ulimit - Core file generation disabled
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: data file size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: open files: 65535 files
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: user processes: 20480 processes
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: cpu time: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Linux transparent hugepage support, enabled="never" defrag="never"
splunk soft fsize unlimited
splunk hard fsize unlimited

0 Karma

tiagofbmm
Influencer

Please let me know if the answer was useful for you. If it was, accept it and upvote. If not, give us more input so we can help you with that

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Customer Survey!

If you use Splunk Observability Cloud, we invite you to share your valuable insights with us through a brief ...

Happy CX Day, Splunk Community!

Happy CX Day, Splunk Community! CX stands for Customer Experience, and today, October 3rd, is CX Day — a ...

.conf23 | Get Your Cybersecurity Defense Analyst Certification in Vegas

We’re excited to announce a new Splunk certification exam being released at .conf23! If you’re going to Las ...