Monitoring Splunk

How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

Splunk Employee
Splunk Employee

I have heard that THP can be problematic for certain applications. I would like to know how this can impact Splunk, and what I need to do about it?

1 Solution

Splunk Employee
Splunk Employee

Some Linux distros have been shipping with THP enabled by default.

See the effects of this on the Splunk documentation here.

The Redhat info here explains 1 method of disabling THP (using grub.conf) as well as providing ways to validate they are disabled.

I like to follow this procedure:
(Each Sys Admin can come up with their own way to pull this off)

I run these two commands on all my systems that are running CentOS/Redhat 6.x or later that are splunk servers.

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

No need to restart splunk
Then to make these changes persistent across reboots I add this to the bottom of my /etc/rc.local

 #disable THP at boot time
if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then
      echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then
    echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

I then validate I am not running things like ktune or tuned (this could actually override the settings you set above)

  chkconfig --list |grep tune
     ktune           0:off   1:off   2:off   3:off   4:off   5:off   6:off
     tuned           0:off   1:off   2:off   3:off   4:off   5:off   6:off

To validate THP is disabled, I run the below three commands, or any variant you choose from here .

cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
       always madvise [never]
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
       always madvise [never]
egrep 'trans|thp' /proc/vmstat  (on this command I validate none of these results are changing)
        nr_anon_transparent_hugepages 2
        thp_fault_alloc 12793
        thp_fault_fallback 18
        thp_collapse_alloc 70
        thp_collapse_alloc_failed 0
        thp_split 2974

One thing to keep in mind, Splunk will log in $SPLUNK_HOME/var/log/splunk/splunkd.log on start up if THP is enabled or disabled

    grep hugetables /opt/splunk/var/log/splunk/splunkd.log
   11-18-2014 08:19:42.052 -0600 INFO  ulimit - Linux transparent hugetables support, enabled="never" defrag="never"

A possible concern with this log entry can be two fold.

  • Because on my system /etc/rc.d/rc3.d/S90splunk is executed before /etc/rc.d/rc3.d/S99local after a reboot the splunkd.log entry will reflect they are enabled. However subsequent splunk restarts would reflect the proper information.

    • This check may not be aware of tuned/ktune running THP.

In summary:

  • Splunk does not have to be restarted after making these changes for the performance gain to be realized.
  • I prefer the rc.local over the grub.conf just because a future kernel upgrade might override my custom stanza.
  • Disabling THP will help improve Splunk performance and recommended to be disabled.

View solution in original post

Path Finder

The official Red Hat link is (jwelsh you forgot a "1" in your link)

0 Karma

Splunk Employee
Splunk Employee

For Redhat / CentOS 7.0 and later here is a possible solution to accommodate systemd:

cat /sys/kernel/mm/transparent_hugepage/enabled  *should see []'s around always
cat /sys/kernel/mm/transparent_hugepage/defrag *should see []'s around always
chmod u+x /etc/rc.d/rc.local
vi /etc/rc.d/rc.local
Place at bottom of file:
              #disable THP at boot time
              if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
                  echo never > /sys/kernel/mm/transparent_hugepage/enabled
            if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
                  echo never > /sys/kernel/mm/transparent_hugepage/defrag
    6. systemctl start rc-local
    7. Repeat steps 1 and 2 and you should now see []'s around never

Splunk Employee
Splunk Employee

It might not work while using tuned. This works as well without skipping tuned:

~# if [ $(grep -E "^\s*GRUB_CMDLINE_LINUX\s*=\s*" /etc/default/grub | grep -Eo "transparent_hugepage\s*=\s*never" | wc -l) -eq 0 ]; then sed -i "s/^(^\s*GRUB_CMDLINE_LINUX\s*=\s*\"[^\"])\s\"\$/\1 transparent_hugepage=never\"/" /etc/default/grub; fi

~# sed -i "s/^(^\s*transparent_hugepages\s*=\s*).*\$/\1never/" /usr/lib/tuned/throughput-performance/tuned.conf

~# reboot

0 Karma

Splunk Employee
Splunk Employee

The more general paths for these controls across different Linux distributions are


The Redhat-specific paths have to do with backporting the feature to older kernels when it was new.

The more general way to persist this type of kernel system setting is via sysctl, but distributions or local practice may have preferred alternatives.

Get Updates on the Splunk Community!

Streamline Data Ingestion With Deployment Server Essentials

REGISTER NOW!Every day the list of sources Admins are responsible for gets bigger and bigger, often making the ...

Remediate Threats Faster and Simplify Investigations With Splunk Enterprise Security ...

REGISTER NOW!Join us for a Tech Talk around our latest release of Splunk Enterprise Security 7.2! We’ll walk ...

Introduction to Splunk AI

WATCH NOWHow are you using AI in Splunk? Whether you see AI as a threat or opportunity, AI is here to stay. ...