Should I disable THP on Linux RHEL 7.2 prior installing Splunk 6.5.0?

[Jrubalcaba]

Does anyone know if this holds valid in RHEL 7.2:

Recently I saw an article regarding Splunk performance and Transparent Huge Pages (THP) in Kernel 2.6
http://docs.splunk.com/Documentation/Splunk/6.5.0/ReleaseNotes/SplunkandTHP

If I am planning to install Splunk 6.5.0 in (distributed search architecture) with two search heads and 3 clustered indexers... Should I disable THP on every Linux RHEL 7.2 Kernel 3.10 instance prior installing Splunk 6.5.0?

[ckurtz]

I reached out to Splunk Engineering (via Support) to confirm this on RHEL7 and Splunk 6.5, and received the following:

Support confirms that "...it is still REQUIRED practice, not simply best practice that THP be disabled."

Here are Engineering's comments regarding the comment from "oschnar":

"His chart is inconclusive. THP doesn't affect/improve indexing rate as much as it severely affects search performance negatively. At low volumes, it doesn't matter, but at higher rates, you'll notice the effect, especially under high search loads/pressure.

AFAIK, the severe performance bug was fixed in later Linux kernels (circa kernel 4.x+, but not back-ported to rhel/centos 7.2) so that madvise is the default, and is reasonably safe to use. However, if the box is performing double duty, cycles will be spent to manage and defrag any use of THP outside of the Splunk process. Since we require dedicated ownership/use of the system, enabling THP for "alien" processes is not recommended.

I had recently confirmed with engineering that we do NOT use THP, but rather we directly allocate/manage hugepages (not the same feature), so disabling THP will be appropriate for any systems running a heavy Splunk process (such as search heads, cluster masters, indexers)."

[ochsner]

Are you sure that this is a good idea for RHEL7.2/x64 systems?

I had the possibility to test this on our box because I had to catch up 3 days of missing data.

When a disabled THP temporary the indexing rate went down quite considerably.
Commands:
switching THP off:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
switching THP back on:
echo always > /sys/kernel/mm/transparent_hugepage/enabled
echo always > /sys/kernel/mm/transparent_hugepage/defrag

I'm wondering if the THP Implementation in RHEL7 is better than it was before?

[Jrubalcaba]

Good point, have you monitor further since turned off THP? RHEL 7.3 will be released in the next couple of months. I don't want to turn it back on.

[rewritex]

Good question. I am at this point of my installation and would like to know of disabling THP is still the recommended course of action for RHEL 7.2 (Maipo). Thanks...

[aaraneta_splunk]

Hi @Jrubalcaba - Looks like you got some good feedback about your question. If one of the answers below has helped, please don't forget to resolve this post by clicking "Accept" below the best answer. If not, please write a comment to share more information. Thank you.

[splk]

Please be aware, the new Health Check is sometimes missleading regarding THP.
THP was disabled on our system, the Health Check shows it as enabled.

(/sys/kernel/mm/redhat_transparent_hugepage/defrag was set to always, and /sys/kernel/mm/redhat_transparent_hugepage/enabled to never which seems to confuse the Health Check).

[richgalloway · Accepted solution]

Yes. Splunk strongly recommends disabling THP on all Linux systems.

[AKG1_old1]

Hi,

definately THP impact the performance of SPLUNK. So it need to be disabled on Linux box. It's not necessary to disable it before installing the SPLUNK. you can disable it after installing the splunk. Just to let you know THP got enabled automatically when we reboot the machine.

Disable the THP:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

to check if THP is been used by SPLUNK
grep -e AnonHugePages /proc/*/smaps | awk '{ if($2>4) print $0} ' | awk -F "/" '{print $0; system("ps -fp " $3)} ' | grep splunk
https://answers.splunk.com/answers/188875/how-do-i-disable-transparent-huge-pages-thp-and-co.html

Thanks
Ankit