Monitoring Splunk
Highlighted

How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Splunk Employee
Splunk Employee

I have heard that THP can be problematic for certain applications. I would like to know how this can impact Splunk, and what I need to do about it?

Highlighted

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Splunk Employee
Splunk Employee

Some Linux distros have been shipping with THP enabled by default.

See the effects of this on the Splunk documentation here.

The Redhat info here explains 1 method of disabling THP (using grub.conf) as well as providing ways to validate they are disabled.

I like to follow this procedure:
(Each Sys Admin can come up with their own way to pull this off)

I run these two commands on all my systems that are running CentOS/Redhat 6.x or later that are splunk servers.

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

No need to restart splunk
Then to make these changes persistent across reboots I add this to the bottom of my /etc/rc.local

 #disable THP at boot time
if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then
      echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then
    echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
fi

I then validate I am not running things like ktune or tuned (this could actually override the settings you set above)

  chkconfig --list |grep tune
     ktune           0:off   1:off   2:off   3:off   4:off   5:off   6:off
     tuned           0:off   1:off   2:off   3:off   4:off   5:off   6:off

To validate THP is disabled, I run the below three commands, or any variant you choose from here .

cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
       always madvise [never]
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
       always madvise [never]
egrep 'trans|thp' /proc/vmstat  (on this command I validate none of these results are changing)
        nr_anon_transparent_hugepages 2
        thp_fault_alloc 12793
        thp_fault_fallback 18
        thp_collapse_alloc 70
        thp_collapse_alloc_failed 0
        thp_split 2974

One thing to keep in mind, Splunk will log in $SPLUNK_HOME/var/log/splunk/splunkd.log on start up if THP is enabled or disabled

    grep hugetables /opt/splunk/var/log/splunk/splunkd.log
   11-18-2014 08:19:42.052 -0600 INFO  ulimit - Linux transparent hugetables support, enabled="never" defrag="never"

A possible concern with this log entry can be two fold.

  • Because on my system /etc/rc.d/rc3.d/S90splunk is executed before /etc/rc.d/rc3.d/S99local after a reboot the splunkd.log entry will reflect they are enabled. However subsequent splunk restarts would reflect the proper information.

    • This check may not be aware of tuned/ktune running THP.

In summary:

  • Splunk does not have to be restarted after making these changes for the performance gain to be realized.
  • I prefer the rc.local over the grub.conf just because a future kernel upgrade might override my custom stanza.
  • Disabling THP will help improve Splunk performance and recommended to be disabled.

View solution in original post

Highlighted

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Splunk Employee
Splunk Employee

The more general paths for these controls across different Linux distributions are

/sys/kernel/mm/transparent_hugepage/enabled
/sys/kernel/mm/transparent_hugepage/defrag

The Redhat-specific paths have to do with backporting the feature to older kernels when it was new.

The more general way to persist this type of kernel system setting is via sysctl, but distributions or local practice may have preferred alternatives.

Highlighted

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Splunk Employee
Splunk Employee

For Redhat / CentOS 7.0 and later here is a possible solution to accommodate systemd:

cat /sys/kernel/mm/transparenthugepage/enabled  *should see []'s around always
cat /sys/kernel/mm/transparent
hugepage/defrag *should see []'s around always
chmod u+x /etc/rc.d/rc.local
vi /etc/rc.d/rc.local
Place at bottom of file:
              #disable THP at boot time
              if test -f /sys/kernel/mm/transparenthugepage/enabled; then
                  echo never > /sys/kernel/mm/transparent
hugepage/enabled
                    fi
            if test -f /sys/kernel/mm/transparenthugepage/defrag; then
                  echo never > /sys/kernel/mm/transparent
hugepage/defrag
                    fi
    6. systemctl start rc-local
    7. Repeat steps 1 and 2 and you should now see []'s around never

Highlighted

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Splunk Employee
Splunk Employee

It might not work while using tuned. This works as well without skipping tuned:

~# if [ $(grep -E "^\sGRUBCMDLINELINUX\s=\s" /etc/default/grub | grep -Eo "transparent_hugepage\s=\snever" | wc -l) -eq 0 ]; then sed -i "s/^(^\sGRUBCMDLINELINUX\s=\s\"[^\"])\s\"\$/\1 transparent_hugepage=never\"/" /etc/default/grub; fi

~# sed -i "s/^(^\stransparent_hugepages\s=\s).\$/\1never/" /usr/lib/tuned/throughput-performance/tuned.conf

~# reboot

0 Karma
Highlighted

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Ultra Champion

Adding yet another version - just with nifty loops and hopefully addressing variants in paths. Perfect for your /etc/rc.local edits...hopefully.

#SPLUNK: disable THP at boot time
THP=`find /sys/kernel/mm/ -name transparent_hugepage -type d | tail -n 1`
for SETTING in "enabled" "defrag";do
    if test -f ${THP}/${SETTING}; then
        echo never > ${THP}/${SETTING}
    fi
done
0 Karma
Highlighted

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Path Finder

The official Red Hat link is https://access.redhat.com/solutions/46111 (jwelsh you forgot a "1" in your link)

0 Karma
Highlighted

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Splunk Employee
Splunk Employee

rc.local above updated as this is a link on some rh6.x but a directory on rh7.x

 #SPLUNK: disable THP at boot time
     THP=`find /sys/kernel/mm/ -name transparent_hugepage \( -type l -o -type d \)| tail -n 1`
     for SETTING in "enabled" "defrag";do
         if test -f ${THP}/${SETTING}; then
             echo never > ${THP}/${SETTING}
         fi
     done
0 Karma
Highlighted

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Ultra Champion

Me gusta.

0 Karma
Highlighted

Re: How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

This needs to be quoted:

echo "never > ${THP}/${SETTING}"
0 Karma