topic Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled? in Monitoring Splunk

How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

jwelch_splunk — Tue, 18 Nov 2014 17:01:54 GMT

I have heard that THP can be problematic for certain applications. I would like to know how this can impact Splunk, and what I need to do about it?

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

jwelch_splunk — Tue, 18 Nov 2014 17:53:35 GMT

Some Linux distros have been shipping with THP enabled by default.

See the effects of this on the Splunk documentation here.

The Redhat info here explains 1 method of disabling THP (using grub.conf) as well as providing ways to validate they are disabled.

I like to follow this procedure:
(Each Sys Admin can come up with their own way to pull this off)

I run these two commands on all my systems that are running CentOS/Redhat 6.x or later that are splunk servers.

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

No need to restart splunk
Then to make these changes persistent across reboots I add this to the bottom of my /etc/rc.local

 #disable THP at boot time
if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then
      echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then
    echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
fi

I then validate I am not running things like ktune or tuned (this could actually override the settings you set above)

  chkconfig --list |grep tune
     ktune           0:off   1:off   2:off   3:off   4:off   5:off   6:off
     tuned           0:off   1:off   2:off   3:off   4:off   5:off   6:off

To validate THP is disabled, I run the below three commands, or any variant you choose from here .

cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
       always madvise [never]
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
       always madvise [never]
egrep 'trans|thp' /proc/vmstat  (on this command I validate none of these results are changing)
        nr_anon_transparent_hugepages 2
        thp_fault_alloc 12793
        thp_fault_fallback 18
        thp_collapse_alloc 70
        thp_collapse_alloc_failed 0
        thp_split 2974

One thing to keep in mind, Splunk will log in $SPLUNK_HOME/var/log/splunk/splunkd.log on start up if THP is enabled or disabled

    grep hugetables /opt/splunk/var/log/splunk/splunkd.log
   11-18-2014 08:19:42.052 -0600 INFO  ulimit - Linux transparent hugetables support, enabled="never" defrag="never"

A possible concern with this log entry can be two fold.

Because on my system /etc/rc.d/rc3.d/S90splunk is executed before /etc/rc.d/rc3.d/S99local after a reboot the splunkd.log entry will reflect they are enabled. However subsequent splunk restarts would reflect the proper information.
- This check may not be aware of tuned/ktune running THP.

In summary:

Splunk does not have to be restarted after making these changes for the performance gain to be realized.
I prefer the rc.local over the grub.conf just because a future kernel upgrade might override my custom stanza.
Disabling THP will help improve Splunk performance and recommended to be disabled.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

mwk1000 — Mon, 28 Sep 2020 19:01:12 GMT

We ran into severe performance degradation and deduced it was memory page handling. An alternative that worked well on our hardware was setting vm.zone_reclaim_mode = 1

This was set to 0 causing memory allocation to occur across nodes in a NUMA architecture. Setting this value forced reclamation of memory assigned to the local node first. With the transient nature of the Splunk search workload processes the impact was stunning.

Server load averages were regularly spiraling out of control many times s over 100 on 12 core / 24 hyperthreaded indexers and searchheads. The symptoms are the same , servers became completely unresponsive for minutes at a time. The root cause may be the same -- poor cross node memory management. We set the parameter and in a matter of a few minutes the load was 2-3 and very hard to drive over 5-6. End user search performance was orders of magnitude faster.

Red Hat Enterprise Linux Server release 6.5 my investigation shows this setting to be CPU specific and some users reported worse performance in our case it was dramatically improved.

Another option to consider when facing similar performance issues.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

dolivasoh — Fri, 03 Apr 2015 15:32:26 GMT

Where did you set this config? I'm seeing similar issues on one of my indexers.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

jrodman — Fri, 05 Jun 2015 13:14:27 GMT

The more general paths for these controls across different Linux distributions are

/sys/kernel/mm/transparent_hugepage/enabled
/sys/kernel/mm/transparent_hugepage/defrag

The Redhat-specific paths have to do with backporting the feature to older kernels when it was new.

The more general way to persist this type of kernel system setting is via sysctl, but distributions or local practice may have preferred alternatives.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

jwelch_splunk — Tue, 29 Sep 2020 07:07:05 GMT

For Redhat / CentOS 7.0 and later here is a possible solution to accommodate systemd:

cat /sys/kernel/mm/transparent_hugepage/enabled *should see []'s around always
cat /sys/kernel/mm/transparent_hugepage/defrag *should see []'s around always
chmod u+x /etc/rc.d/rc.local
vi /etc/rc.d/rc.local
Place at bottom of file:
#disable THP at boot time
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
6. systemctl start rc-local
7. Repeat steps 1 and 2 and you should now see []'s around never

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

apfender_splunk — Tue, 29 Sep 2020 07:17:07 GMT

It might not work while using tuned. This works as well without skipping tuned:

~# if [ $(grep -E "^\s*GRUB_CMDLINE_LINUX\s*=\s*" /etc/default/grub | grep -Eo "transparent_hugepage\s*=\s*never" | wc -l) -eq 0 ]; then sed -i "s/^(^\s*GRUB_CMDLINE_LINUX\s*=\s*\"[^\"])\s\"\$/\1 transparent_hugepage=never\"/" /etc/default/grub; fi

~# sed -i "s/^(^\s*transparent_hugepages\s*=\s*).*\$/\1never/" /usr/lib/tuned/throughput-performance/tuned.conf

~# reboot

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

sloshburch — Sun, 20 Sep 2015 17:57:09 GMT

Adding yet another version - just with nifty loops and hopefully addressing variants in paths. Perfect for your /etc/rc.local edits...hopefully.

#SPLUNK: disable THP at boot time
THP=`find /sys/kernel/mm/ -name transparent_hugepage -type d | tail -n 1`
for SETTING in "enabled" "defrag";do
    if test -f ${THP}/${SETTING}; then
        echo never > ${THP}/${SETTING}
    fi
done

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

giorgio_adami_m — Thu, 14 Jan 2016 11:03:55 GMT

The official Red Hat link is https://access.redhat.com/solutions/46111 (jwelsh you forgot a "1" in your link)

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

jrodman — Tue, 29 Sep 2020 08:30:55 GMT

Syntax like vm.zone_reclaim_mode is a sysctl value. There will be an equivalent virtual file in proc with slightly different naming.

The existence of these values can be kernel version dependent, but the referred system is RHEL6.5, so typically it will be availble there and on newer systems.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

seunomosowon — Fri, 15 Apr 2016 08:26:01 GMT

Hi mwk1000,

Mind if I ask you what hardware you were running the indexers on when you had to set this?

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

jrodman — Fri, 15 Apr 2016 08:59:26 GMT

This problem is generic to all Linux kernels for the last several years.
It becomes really catastrophic as memory becomes closer to exhausted (e.g over 80% in use and rising), because of the thrashing to defragment pages. However, the work to simply map in transparent huge pages for relatively short lived search processes is always a net loss.

It's a no-brainer to always turn these off in all Linux installs.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

cmeo — Tue, 13 Dec 2016 06:30:11 GMT

Putting it in rc.local is too late. Using the legacy sysV init script which splunk enable boot-start deploys, it's already started by then.

What I do is modify /etc/init.d/splunk as follows:
Add function:

RETVAL=0 #existing

# disable hugepages

disable_huge() {
  echo "disabling huge page support"
  if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
     echo never > /sys/kernel/mm/transparent_hugepage/enabled
  fi
  if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
     echo never > /sys/kernel/mm/transparent_hugepage/defrag
  fi
}

Use it:

case "$1" in
  start)
    disable_huge
    splunk_start
    ;;
  stop)
    splunk_stop
    ;;
  restart)
    disable_huge
    splunk_restart
    ;;

Tried all the other things here and in places like stackoverflow, seems to be the only thing that actually works.
Yes we should all be using systemd methodology everywhere, but you know what? CBF learning yet another OS startup gadget, of which the woods seem to be full.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

jwelch_splunk — Tue, 13 Dec 2016 13:28:07 GMT

As I mentioned in my original posting, THP can be disabled on the fly "without restarting splunk", and you are correct in normal SysV we would be disabling THP with rc.local after splunk has started, as I mentioned in the article as well. However it is only enabled for about 1 second before rc.local turns it back off.

Also using grub.conf is an option as well.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

vliggio — Tue, 29 Sep 2020 12:13:27 GMT

/sys/kernel/mm/redhat_transparent_hugepage/defrag does NOT need to be set to never if /sys/kernel/mm/redhat_transparent_hugepage/enabled is set to never. The system will not try to defrag something that is already off. This is documented in the RedHat guide to turning off THP.

Also, doing it via grub will survive kernel upgrades. By default, all options are carried over to the new kernel lines in grub.conf.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

cmeo — Fri, 30 Dec 2016 00:37:47 GMT

The grub.conf solution, as someone else already pointed out, can be overwritten by a kernel update.
What might help is a proper systemd-type service to do this as a dependency for splunk before it starts (and a lot of other software doesn't like THP either so this would be widely useful).
Alternatively some proper THP controls created upstream in the distros, but I'm not holding my breath. hugeadm in ubuntu is a step in the right direction but that doesn't work properly either as I discovered when I tried to use it. It's not sticky IIRC or perhaps it was some other problem--I don't recall. And it doesn't help you with any other distro anyway.
Yet another possiblity would be for splunk to actually address the problem and update the boot script apparatus. The ulimit stuff is out of date too. The only thing I've found that actually worked is here:
http://stackoverflow.com/questions/39506149/ubuntu-16-04-systemd-redis-issues-with-ulimit
and this is a lot of messing around to achieve what used to be easy but now isn't.
This still leaves you with a problem if someone runs $SPLUNK_HOME/bin/splunk start or restart without a wrapper script. All in all quite exasperating and I freely admit I've created an ugly hack.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

vliggio — Fri, 30 Dec 2016 14:59:31 GMT

I've never had a grub.conf overwritten by a kernel update in 20 years using any RedHat derived distro. Maybe other ones don't behave well (I refuse to use ubuntu). I believe any modifications are probably happening through ktune or tuned, neither of which I use.

It would definitely be better if Splunk took care of the configs as part of the install, at least for people who aren't so adept at creating their own configs. Or just don't start up until the underlying configs are correct. Why compromise the performance of Splunk for things that are known like THP and open file settings?

There's no need to do the defrag setting when THP is off no matter how you turn off THP, since there's nothing to defrag.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

maraman_splunk — Tue, 03 Jan 2017 14:22:13 GMT

rc.local above updated as this is a link on some rh6.x but a directory on rh7.x

 #SPLUNK: disable THP at boot time
     THP=`find /sys/kernel/mm/ -name transparent_hugepage \( -type l -o -type d \)| tail -n 1`
     for SETTING in "enabled" "defrag";do
         if test -f ${THP}/${SETTING}; then
             echo never > ${THP}/${SETTING}
         fi
     done

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

sloshburch — Wed, 04 Jan 2017 13:41:18 GMT

Me gusta.

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

kamal_jagga — Fri, 06 Jan 2017 23:47:31 GMT

Can anyone suggest any easy way to validate whether disabling THP improved the performance ?