Solved: Re: How do I disable Transparent Huge Pages (THP) ... - Page 3

jwelch_splunk · ‎11-18-2014

I have heard that THP can be problematic for certain applications. I would like to know how this can impact Splunk, and what I need to do about it?

jwelch_splunk · ‎11-18-2014

Some Linux distros have been shipping with THP enabled by default.

See the effects of this on the Splunk documentation here.

The Redhat info here explains 1 method of disabling THP (using grub.conf) as well as providing ways to validate they are disabled.

I like to follow this procedure:
(Each Sys Admin can come up with their own way to pull this off)

I run these two commands on all my systems that are running CentOS/Redhat 6.x or later that are splunk servers.

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

No need to restart splunk
Then to make these changes persistent across reboots I add this to the bottom of my /etc/rc.local

 #disable THP at boot time
if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then
      echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then
    echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
fi

I then validate I am not running things like ktune or tuned (this could actually override the settings you set above)

  chkconfig --list |grep tune
     ktune           0:off   1:off   2:off   3:off   4:off   5:off   6:off
     tuned           0:off   1:off   2:off   3:off   4:off   5:off   6:off

To validate THP is disabled, I run the below three commands, or any variant you choose from here .

cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
       always madvise [never]
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
       always madvise [never]
egrep 'trans|thp' /proc/vmstat  (on this command I validate none of these results are changing)
        nr_anon_transparent_hugepages 2
        thp_fault_alloc 12793
        thp_fault_fallback 18
        thp_collapse_alloc 70
        thp_collapse_alloc_failed 0
        thp_split 2974

One thing to keep in mind, Splunk will log in $SPLUNK_HOME/var/log/splunk/splunkd.log on start up if THP is enabled or disabled

    grep hugetables /opt/splunk/var/log/splunk/splunkd.log
   11-18-2014 08:19:42.052 -0600 INFO  ulimit - Linux transparent hugetables support, enabled="never" defrag="never"

A possible concern with this log entry can be two fold.

Because on my system /etc/rc.d/rc3.d/S90splunk is executed before /etc/rc.d/rc3.d/S99local after a reboot the splunkd.log entry will reflect they are enabled. However subsequent splunk restarts would reflect the proper information.
- This check may not be aware of tuned/ktune running THP.

In summary:

Splunk does not have to be restarted after making these changes for the performance gain to be realized.
I prefer the rc.local over the grub.conf just because a future kernel upgrade might override my custom stanza.
Disabling THP will help improve Splunk performance and recommended to be disabled.

View solution in original post

giorgio_adami_m · ‎01-14-2016

The official Red Hat link is https://access.redhat.com/solutions/46111 (jwelsh you forgot a "1" in your link)

jwelch_splunk · ‎08-24-2015

For Redhat / CentOS 7.0 and later here is a possible solution to accommodate systemd:

cat /sys/kernel/mm/transparent_hugepage/enabled *should see []'s around always
cat /sys/kernel/mm/transparent_hugepage/defrag *should see []'s around always
chmod u+x /etc/rc.d/rc.local
vi /etc/rc.d/rc.local
Place at bottom of file:
#disable THP at boot time
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
6. systemctl start rc-local
7. Repeat steps 1 and 2 and you should now see []'s around never

apfender_splunk · ‎09-17-2015

It might not work while using tuned. This works as well without skipping tuned:

~# if [ $(grep -E "^\s*GRUB_CMDLINE_LINUX\s*=\s*" /etc/default/grub | grep -Eo "transparent_hugepage\s*=\s*never" | wc -l) -eq 0 ]; then sed -i "s/^(^\s*GRUB_CMDLINE_LINUX\s*=\s*\"[^\"])\s\"\$/\1 transparent_hugepage=never\"/" /etc/default/grub; fi

~# sed -i "s/^(^\s*transparent_hugepages\s*=\s*).*\$/\1never/" /usr/lib/tuned/throughput-performance/tuned.conf

~# reboot

jrodman · ‎06-05-2015

The more general paths for these controls across different Linux distributions are

/sys/kernel/mm/transparent_hugepage/enabled
/sys/kernel/mm/transparent_hugepage/defrag

The Redhat-specific paths have to do with backporting the feature to older kernels when it was new.

The more general way to persist this type of kernel system setting is via sysctl, but distributions or local practice may have preferred alternatives.

How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Announcing Scheduled Export GA for Dashboard Studio

Extending Observability Content to Splunk Cloud

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!