https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358505#M13249 <P>Hi,<BR /> Today morning 2 search heads out of 3 from cluster went down. When i checked it killed by OS with message 'out of memory' in /var/log/message, but system had enough memory at that time when splunk process got killed by OS [around 38% was free]. In splunkd.log file i did not find any error messages. Can any one please let me know how to get the root cause of this issue and fix it. This already happen 3-4 times.<BR /> Thanks in advance.</P> <P>thanks,<BR /> Shashank Soni.</P> Tue, 02 May 2017 16:03:09 GMT Yod_ssoni 2017-05-02T16:03:09Z https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358505#M13249 <P>Hi,<BR /> Today morning 2 search heads out of 3 from cluster went down. When i checked it killed by OS with message 'out of memory' in /var/log/message, but system had enough memory at that time when splunk process got killed by OS [around 38% was free]. In splunkd.log file i did not find any error messages. Can any one please let me know how to get the root cause of this issue and fix it. This already happen 3-4 times.<BR /> Thanks in advance.</P> <P>thanks,<BR /> Shashank Soni.</P> Tue, 02 May 2017 16:03:09 GMT https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358505#M13249 Yod_ssoni 2017-05-02T16:03:09Z https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358506#M13250 <P>Have you checked your ulimits ? </P> Tue, 02 May 2017 19:03:42 GMT https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358506#M13250 klaxdal 2017-05-02T19:03:42Z https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358507#M13251 <P>Ulimits can trigger OOM killer too from what I understand. Up voting.</P> Tue, 02 May 2017 23:01:50 GMT https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358507#M13251 jkat54 2017-05-02T23:01:50Z https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358508#M13252 <P><A href="http://www.georgestarcher.com/splunk-ulimits-and-you/">http://www.georgestarcher.com/splunk-ulimits-and-you/</A></P> Tue, 02 May 2017 23:03:54 GMT https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358508#M13252 jkat54 2017-05-02T23:03:54Z https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358509#M13253 <P>THP being enabled is the #1 reason for poor Splunk RAM management. Run a health check from your MC and see if everything is setup correctly. This will check for ulimits, too.</P> Thu, 28 Jun 2018 15:17:31 GMT https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358509#M13253 woodcock 2018-06-28T15:17:31Z https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358510#M13254 <P>Just wanted to add this link describing THP's impact on memory to woodcock's answer.</P> <P><A href="http://docs.splunk.com/Documentation/Splunk/7.1.1/ReleaseNotes/SplunkandTHP">http://docs.splunk.com/Documentation/Splunk/7.1.1/ReleaseNotes/SplunkandTHP</A></P> <P>Hope this helps.</P> Thu, 28 Jun 2018 15:43:40 GMT https://community.splunk.com/t5/Deployment-Architecture/Splunk-process-killed-by-OS-out-of-memory-in-Search-head/m-p/358510#M13254 lacastillo 2018-06-28T15:43:40Z