I have heard that THP can be problematic for certain applications. I would like to know how this can impact Splunk, and what I need to do about it?
Some Linux distros have been shipping with THP enabled by default.
See the effects of this on the Splunk documentation here.
The Redhat info here explains 1 method of disabling THP (using grub.conf) as well as providing ways to validate they are disabled.
I like to follow this procedure:
(Each Sys Admin can come up with their own way to pull this off)
I run these two commands on all my systems that are running CentOS/Redhat 6.x or later that are splunk servers.
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
No need to restart splunk
Then to make these changes persistent across reboots I add this to the bottom of my /etc/rc.local
#disable THP at boot time if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled fi if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag fi
I then validate I am not running things like ktune or tuned (this could actually override the settings you set above)
chkconfig --list |grep tune ktune 0:off 1:off 2:off 3:off 4:off 5:off 6:off tuned 0:off 1:off 2:off 3:off 4:off 5:off 6:off
To validate THP is disabled, I run the below three commands, or any variant you choose from here .
cat /sys/kernel/mm/redhat_transparent_hugepage/defrag always madvise [never] cat /sys/kernel/mm/redhat_transparent_hugepage/enabled always madvise [never] egrep 'trans|thp' /proc/vmstat (on this command I validate none of these results are changing) nr_anon_transparent_hugepages 2 thp_fault_alloc 12793 thp_fault_fallback 18 thp_collapse_alloc 70 thp_collapse_alloc_failed 0 thp_split 2974
One thing to keep in mind, Splunk will log in $SPLUNK_HOME/var/log/splunk/splunkd.log on start up if THP is enabled or disabled
grep hugetables /opt/splunk/var/log/splunk/splunkd.log 11-18-2014 08:19:42.052 -0600 INFO ulimit - Linux transparent hugetables support, enabled="never" defrag="never"
A possible concern with this log entry can be two fold.
Because on my system /etc/rc.d/rc3.d/S90splunk is executed before /etc/rc.d/rc3.d/S99local after a reboot the splunkd.log entry will reflect they are enabled. However subsequent splunk restarts would reflect the proper information.
The official Red Hat link is https://access.redhat.com/solutions/46111 (jwelsh you forgot a "1" in your link)
For Redhat / CentOS 7.0 and later here is a possible solution to accommodate systemd:
cat /sys/kernel/mm/transparent_hugepage/enabled *should see 's around always
cat /sys/kernel/mm/transparent_hugepage/defrag *should see 's around always
chmod u+x /etc/rc.d/rc.local
Place at bottom of file:
#disable THP at boot time
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
6. systemctl start rc-local
7. Repeat steps 1 and 2 and you should now see 's around never
It might not work while using tuned. This works as well without skipping tuned:
~# if [ $(grep -E "^\s*GRUB_CMDLINE_LINUX\s*=\s*" /etc/default/grub | grep -Eo "transparent_hugepage\s*=\s*never" | wc -l) -eq 0 ]; then sed -i "s/^(^\s*GRUB_CMDLINE_LINUX\s*=\s*\"[^\"])\s\"\$/\1 transparent_hugepage=never\"/" /etc/default/grub; fi
~# sed -i "s/^(^\s*transparent_hugepages\s*=\s*).*\$/\1never/" /usr/lib/tuned/throughput-performance/tuned.conf
The more general paths for these controls across different Linux distributions are
The Redhat-specific paths have to do with backporting the feature to older kernels when it was new.
The more general way to persist this type of kernel system setting is via sysctl, but distributions or local practice may have preferred alternatives.