Deployment Architecture

Should I disable THP on Linux RHEL 7.2 prior installing Splunk 6.5.0?

Explorer

Does anyone know if this holds valid in RHEL 7.2:

Recently I saw an article regarding Splunk performance and Transparent Huge Pages (THP) in Kernel 2.6
http://docs.splunk.com/Documentation/Splunk/6.5.0/ReleaseNotes/SplunkandTHP

If I am planning to install Splunk 6.5.0 in (distributed search architecture) with two search heads and 3 clustered indexers... Should I disable THP on every Linux RHEL 7.2 Kernel 3.10 instance prior installing Splunk 6.5.0?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Yes. Splunk strongly recommends disabling THP on all Linux systems.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

Path Finder

I reached out to Splunk Engineering (via Support) to confirm this on RHEL7 and Splunk 6.5, and received the following:

Support confirms that "...it is still REQUIRED practice, not simply best practice that THP be disabled."

Here are Engineering's comments regarding the comment from "oschnar":

"His chart is inconclusive. THP doesn't affect/improve indexing rate as much as it severely affects search performance negatively. At low volumes, it doesn't matter, but at higher rates, you'll notice the effect, especially under high search loads/pressure.

AFAIK, the severe performance bug was fixed in later Linux kernels (circa kernel 4.x+, but not back-ported to rhel/centos 7.2) so that madvise is the default, and is reasonably safe to use. However, if the box is performing double duty, cycles will be spent to manage and defrag any use of THP outside of the Splunk process. Since we require dedicated ownership/use of the system, enabling THP for "alien" processes is not recommended.

I had recently confirmed with engineering that we do NOT use THP, but rather we directly allocate/manage hugepages (not the same feature), so disabling THP will be appropriate for any systems running a heavy Splunk process (such as search heads, cluster masters, indexers)."

Engager

Are you sure that this is a good idea for RHEL7.2/x64 systems?

I had the possibility to test this on our box because I had to catch up 3 days of missing data.

Image of the indexing rate graph

When a disabled THP temporary the indexing rate went down quite considerably.
Commands:
switching THP off:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
switching THP back on:
echo always > /sys/kernel/mm/transparent_hugepage/enabled
echo always > /sys/kernel/mm/transparent_hugepage/defrag

I'm wondering if the THP Implementation in RHEL7 is better than it was before?

Explorer

Good point, have you monitor further since turned off THP? RHEL 7.3 will be released in the next couple of months. I don't want to turn it back on.

0 Karma

Contributor

Good question. I am at this point of my installation and would like to know of disabling THP is still the recommended course of action for RHEL 7.2 (Maipo). Thanks...

0 Karma

Splunk Employee
Splunk Employee

Hi @Jrubalcaba - Looks like you got some good feedback about your question. If one of the answers below has helped, please don't forget to resolve this post by clicking "Accept" below the best answer. If not, please write a comment to share more information. Thank you.

0 Karma

Communicator

Please be aware, the new Health Check is sometimes missleading regarding THP.
THP was disabled on our system, the Health Check shows it as enabled.

(/sys/kernel/mm/redhat_transparent_hugepage/defrag was set to always, and /sys/kernel/mm/redhat_transparent_hugepage/enabled to never which seems to confuse the Health Check).

SplunkTrust
SplunkTrust

Yes. Splunk strongly recommends disabling THP on all Linux systems.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

Builder

Hi,

definately THP impact the performance of SPLUNK. So it need to be disabled on Linux box. It's not necessary to disable it before installing the SPLUNK. you can disable it after installing the splunk. Just to let you know THP got enabled automatically when we reboot the machine.

Disable the THP:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

to check if THP is been used by SPLUNK
grep -e AnonHugePages /proc/*/smaps | awk '{ if($2>4) print $0} ' | awk -F "/" '{print $0; system("ps -fp " $3)} ' | grep splunk
https://answers.splunk.com/answers/188875/how-do-i-disable-transparent-huge-pages-thp-and-co.html

Thanks
Ankit

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!