HIgh Physcial Memory Usage in Splunk searchhead VM

[kartreddy4]

Hi , I am currently facing this High Physical VM usage alert in splunk adhoc searchhead which is in VM environment (just did the HealthCheck). The specs are
Memory Used (MB) 17592186037778.105
Physical Memory Installed (MB) 15870.715
Memory used (%) 110846839841.7%

The THP aren't disabled and open files are 4096 against the recommended 8192. What could be the reason for this excessive memory usage and how to control it. Help would be highly appreciated.

Regards,
KJ

[deepashri_123]

@kartreddy4,
You can try referring this doc:
http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Troubleshootmemoryusage

[tiagofbmm]

Well not having the THP and ulimits properly set could be the main issue

Check If this system is running Splunk as a user called "splunk," add the following to the bottom of the /etc/security/limits.conf file:

Should look like this your cat /opt/splunk/var/log/splunk/splunkd.log | grep ulimit

03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: virtual address space size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: data segment size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: resident memory size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: core file size: 0 bytes
03-01-2017 22:08:01.857 +0000 WARN ulimit - Core file generation disabled
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: data file size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: open files: 65535 files
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: user processes: 20480 processes
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: cpu time: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Linux transparent hugepage support, enabled="never" defrag="never"
splunk soft fsize unlimited
splunk hard fsize unlimited

[tiagofbmm]

Please let me know if the answer was useful for you. If it was, accept it and upvote. If not, give us more input so we can help you with that