topic Re: Splunk complains about Transparent Huge Pages defrag, sysadmin says it does not matter. Which is true? in Deployment Architecture

Splunk complains about Transparent Huge Pages defrag, sysadmin says it does not matter. Which is true?

wrangler2x — Thu, 27 Apr 2017 22:31:56 GMT

I noticed that Splunk was complaining about Transparent Huge Pages being enabled on a new platform we just migrated to. You'd see this in splunkd.log:

04-26-2017 16:03:55.871 -0700 INFO ulimit - Linux transparent hugepage support, enabled="always" defrag="always"
04-26-2017 16:03:55.871 -0700 WARN ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.

The system is RHEL 7. The administrator of the system has set /sys/kernel/mm/transparent_hugepage/enabled to [never] but has left /sys/kernel/mm/transparent_hugepage/defrag as [always] with this claim:

The one note about this approach is that you might be mislead about THP status is you look at the defrag status which is set to always. This actually doesn't matter when THP is disabled since no defrag will ever be performed. You can verify THP status with this command "cat /sys/kernel/mm/transparent_hugepage/enabled"

My question: is he right about that? And if so, why is Splunk still saying "This configuration of transparent hugepages is known to cause serious runtime problems with Splunk." ?

If he is wrong, what can I show him to make him believe he has to set these both to [never]?

Re: Splunk complains about Transparent Huge Pages defrag, sysadmin says it does not matter. Which is true?

adonio — Thu, 27 Apr 2017 23:46:34 GMT

check this answer:
https://answers.splunk.com/answers/188875/how-do-i-disable-transparent-huge-pages-thp-and-co.html
explains it in detail

Re: Splunk complains about Transparent Huge Pages defrag, sysadmin says it does not matter. Which is true?

muebel — Thu, 27 Apr 2017 23:50:46 GMT

Hi wrangler2x,

It certainly does matter. I work with a multi terrabyte installation, and I can tell you that THP needs to be completely disabled for any system in the splunk infrastructure. In particular the search heads and indexers. If you ever file a support ticket with splunk for performance issues, you'll find this is one of the first things they check, with good reason. I've seen it very clearly resolve performance issues in the past.

Please let me know if this answers your question! 😄

Re: Splunk complains about Transparent Huge Pages defrag, sysadmin says it does not matter. Which is true?

suarezry — Fri, 28 Apr 2017 14:30:57 GMT

Your sysadmin is right. He has disabled THP so the THPdefrag option does not matter:
https://access.redhat.com/solutions/46111

NOTE: Some third party application install scripts check value of above files and complain even if THP is disabled at boot time using transparent_hugepage=never, this is due to the fact when THP is disabled at boot time, the value of /sys/kernel/mm/redhat_transparent_hugepage/defrag will not be changed, however this is expected and system will never go in THP defragmentation code path when it is disabled at boot and THP defrag need not to be disabled separately.

But with that said what's the big deal? Just set the defrag to 'never' and be done with it!

Re: Splunk complains about Transparent Huge Pages defrag, sysadmin says it does not matter. Which is true?

suarezry — Fri, 28 Apr 2017 14:32:44 GMT

The post says that the sysadmin disabled THP so OP's is asking if the THPdefrag option still matters. It does not.

Re: Splunk complains about Transparent Huge Pages defrag, sysadmin says it does not matter. Which is true?

wrangler2x — Fri, 28 Apr 2017 17:12:15 GMT

Yes, I shared this document with him. He's going by a Red Hat page I don't have access to.

Re: Splunk complains about Transparent Huge Pages defrag, sysadmin says it does not matter. Which is true?

wrangler2x — Fri, 28 Apr 2017 17:14:38 GMT

I don't know what the big deal is. Ego? I've created this support request:

Case Number 478790 -- Subject: Transparent Huge Pages aka "transparent hugepages" Splunk warning

Re: Splunk complains about Transparent Huge Pages defrag, sysadmin says it does not matter. Which is true?

wrangler2x — Tue, 29 Sep 2020 13:52:59 GMT

From Splunk Support Fri Apr 28 11:24:25 2017
Date: Fri, 28 Apr 2017 17:21:43 +0000 (GMT)
From: Splunk Support
To:
Subject: Case #478790 Transparent Huge Pages aka "transparent hugepages"
Splunk warning [ ref:_00D409oyL._5003317y65P:ref ]

Hello Garrett -

Thank you for contacting Splunk Technical Support. The second post-er is correct; it doesn't matter if defrag is set to never - if you're OK with seeing WARN'ings in the log.

We advise to turn off both. Our documentation points to an Answers page that has been verified by Splunk as correct, and it specifically describes turning off both.

https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP

https://answers.splunk.com/answers/188875/how-do-i-disable-transparent-huge-pages-thp-and-co.html

So, run with that and you'll be good to go!

Regards,

Eric Ryder
Splunk Technical Support
US Phone: 55-775-8657/855-SPLUNKS
Outside US: http://www.splunk.com/view/contact-us/SP-CAAAHJ6
Open a case: p://support.splunk.com/

Case: 478790
Status: Open
Priority Level: P3

[ ref:_00D409oyL._5003317y65P:ref ]

From Splunk Support Fri Apr 28 11:24:30 2017
Date: Fri, 28 Apr 2017 17:28:12 +0000 (GMT)
From: Splunk Support
To:
Subject: Case #478790 Transparent Huge Pages aka "transparent hugepages"
Splunk warning [ ref:_00D409oyL._5003317y65P:ref ]

p.s. To clarify my last email: Leaving defrag at "always" does not have a performance impact, but you will see the warning in the log. We don't consider this a bug, because we ask that it also be turned off, per the verified Answers post with instructions.

I hope that's more clear.

Cheers,

Eric Ryder
Splunk Technical Support
US Phone: 55-775-8657/855-SPLUNKS
Outside US: http://www.splunk.com/view/contact-us/SP-CAAAHJ6
Open a case: p://support.splunk.com/