Deployment Architecture

Should I disable THP on Linux RHEL 7.2 prior installing Splunk 6.5.0?

Jrubalcaba
Explorer

Does anyone know if this holds valid in RHEL 7.2:

Recently I saw an article regarding Splunk performance and Transparent Huge Pages (THP) in Kernel 2.6
http://docs.splunk.com/Documentation/Splunk/6.5.0/ReleaseNotes/SplunkandTHP

If I am planning to install Splunk 6.5.0 in (distributed search architecture) with two search heads and 3 clustered indexers... Should I disable THP on every Linux RHEL 7.2 Kernel 3.10 instance prior installing Splunk 6.5.0?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes. Splunk strongly recommends disabling THP on all Linux systems.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

ckurtz
Path Finder

I reached out to Splunk Engineering (via Support) to confirm this on RHEL7 and Splunk 6.5, and received the following:

Support confirms that "...it is still REQUIRED practice, not simply best practice that THP be disabled."

Here are Engineering's comments regarding the comment from "oschnar":

"His chart is inconclusive. THP doesn't affect/improve indexing rate as much as it severely affects search performance negatively. At low volumes, it doesn't matter, but at higher rates, you'll notice the effect, especially under high search loads/pressure.

AFAIK, the severe performance bug was fixed in later Linux kernels (circa kernel 4.x+, but not back-ported to rhel/centos 7.2) so that madvise is the default, and is reasonably safe to use. However, if the box is performing double duty, cycles will be spent to manage and defrag any use of THP outside of the Splunk process. Since we require dedicated ownership/use of the system, enabling THP for "alien" processes is not recommended.

I had recently confirmed with engineering that we do NOT use THP, but rather we directly allocate/manage hugepages (not the same feature), so disabling THP will be appropriate for any systems running a heavy Splunk process (such as search heads, cluster masters, indexers)."

ochsner
Engager

Are you sure that this is a good idea for RHEL7.2/x64 systems?

I had the possibility to test this on our box because I had to catch up 3 days of missing data.

Image of the indexing rate graph

When a disabled THP temporary the indexing rate went down quite considerably.
Commands:
switching THP off:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
switching THP back on:
echo always > /sys/kernel/mm/transparent_hugepage/enabled
echo always > /sys/kernel/mm/transparent_hugepage/defrag

I'm wondering if the THP Implementation in RHEL7 is better than it was before?

Jrubalcaba
Explorer

Good point, have you monitor further since turned off THP? RHEL 7.3 will be released in the next couple of months. I don't want to turn it back on.

0 Karma

rewritex
Contributor

Good question. I am at this point of my installation and would like to know of disabling THP is still the recommended course of action for RHEL 7.2 (Maipo). Thanks...

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

Hi @Jrubalcaba - Looks like you got some good feedback about your question. If one of the answers below has helped, please don't forget to resolve this post by clicking "Accept" below the best answer. If not, please write a comment to share more information. Thank you.

0 Karma

splk
Communicator

Please be aware, the new Health Check is sometimes missleading regarding THP.
THP was disabled on our system, the Health Check shows it as enabled.

(/sys/kernel/mm/redhat_transparent_hugepage/defrag was set to always, and /sys/kernel/mm/redhat_transparent_hugepage/enabled to never which seems to confuse the Health Check).

richgalloway
SplunkTrust
SplunkTrust

Yes. Splunk strongly recommends disabling THP on all Linux systems.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

agoyal
Builder

Hi,

definately THP impact the performance of SPLUNK. So it need to be disabled on Linux box. It's not necessary to disable it before installing the SPLUNK. you can disable it after installing the splunk. Just to let you know THP got enabled automatically when we reboot the machine.

Disable the THP:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

to check if THP is been used by SPLUNK
grep -e AnonHugePages /proc/*/smaps | awk '{ if($2>4) print $0} ' | awk -F "/" '{print $0; system("ps -fp " $3)} ' | grep splunk
https://answers.splunk.com/answers/188875/how-do-i-disable-transparent-huge-pages-thp-and-co.html

Thanks
Ankit

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!