Some Linux distros have been shipping with THP enabled by default.
See the effects of this on the Splunk documentation here.
The Redhat info here explains 1 method of disabling THP (using grub.conf) as well as providing ways to validate they are disabled.
I like to follow this procedure:
(Each Sys Admin can come up with their own way to pull this off)
I run these two commands on all my systems that are running CentOS/Redhat 6.x or later that are splunk servers.
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
No need to restart splunk
Then to make these changes persistent across reboots I add this to the bottom of my /etc/rc.local
#disable THP at boot time
if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
fi
I then validate I am not running things like ktune or tuned (this could actually override the settings you set above)
chkconfig --list |grep tune
ktune 0:off 1:off 2:off 3:off 4:off 5:off 6:off
tuned 0:off 1:off 2:off 3:off 4:off 5:off 6:off
To validate THP is disabled, I run the below three commands, or any variant you choose from here .
cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
always madvise [never]
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always madvise [never]
egrep 'trans|thp' /proc/vmstat (on this command I validate none of these results are changing)
nr_anon_transparent_hugepages 2
thp_fault_alloc 12793
thp_fault_fallback 18
thp_collapse_alloc 70
thp_collapse_alloc_failed 0
thp_split 2974
One thing to keep in mind, Splunk will log in $SPLUNK_HOME/var/log/splunk/splunkd.log on start up if THP is enabled or disabled
grep hugetables /opt/splunk/var/log/splunk/splunkd.log
11-18-2014 08:19:42.052 -0600 INFO ulimit - Linux transparent hugetables support, enabled="never" defrag="never"
A possible concern with this log entry can be two fold.
In summary:
- Splunk does not have to be restarted after making these changes for the performance gain to be realized.
- I prefer the rc.local over the grub.conf just because a future kernel upgrade might override my custom stanza.
- Disabling THP will help improve Splunk performance and recommended to be disabled.
