Indexers Crashing from low memory, any way to prevent his?

[daniel333]

All,

My CentOS7 indexers in a cluster are crashing prety regularly now. /var/log/messages shows OOM errors and I can confirm we're pretty high on jobs being dispatched to Splunk. Is there any way to limit memory per search? I see some notes say limits.conf on the Search head, but my issue is the indexers. Same config?

[esix_splunk]

First, do you have THP disabled? There are some pretty major bugs with Redhat and CentOS regarding Transparent Huge Pages and Splunk memory consumption. Disable this..

More reading here : https://docs.splunk.com/Documentation/Splunk/7.0.2/ReleaseNotes/SplunkandTHP

Aside from that, have you opened a support case with Splunk? There is some tuning you can do in limits.conf on both the SH and the IDX, however, its best if you get those parameters recommended via official support. They can provide a more comprehensive view of how these may effect your system.