Getting Data In

Turn THP off on Universal Forwarder?

mfrost8
Builder

I get the whole thing about turning off THP on Splunk Enterprise instances per https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP and many other places.

However, everything I've seen refers pretty specifically to Splunk Enterprise instances. I would have assumed that this wouldn't be necessary on a universal forwarder, yet on our late-model UF installations, I always see:

04-24-2017 07:33:50.452 -0500 WARN  ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.

Being, the same message as you'd get on a Splunk Enterprise instance, it seems that Splunk wants this turned off on universal forwarder installations too? I'm somewhat less comfortable with that on servers whose function is exclusively Splunk as the impact of THP is unknown to me.

Or perhaps is the existence of this warning on a universal forwarder a bug of some sort? I'm pretty sure I've seen it there for quite a few versions now.

Thanks!

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi mfrost8,
I didn't experienced problems related to THP on forwarders because "On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance", see (https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP).
In addition in http://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements they speak about Splunk Enterprise and not about Splunk Forwarders.

Bye.
Giuseppe

View solution in original post

0 Karma

mfrost8
Builder

I opened a case with Splunk on this. Despite the same dire warning about THP, it's probably not quite as essential on a universal forwarder. One would need to look at the specifics of THP and the generalizations about how it impacts applications.

It seems that Splunk could benefit from it when you've got a lot of open files. Perhaps less essential if you've only got a few. It's not likely to be impactful to a non-Splunk app, unless it's doing operations on large files on a system with very little memory.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi mfrost8,
I didn't experienced problems related to THP on forwarders because "On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance", see (https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP).
In addition in http://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements they speak about Splunk Enterprise and not about Splunk Forwarders.

Bye.
Giuseppe

0 Karma

mfrost8
Builder

Right, but then why the warning in the universal forwarder logs?

0 Karma

gcusello
SplunkTrust
SplunkTrust

I have many Red hat Forwarders but I haven't this message.
Are you using Universal or Heavy Forwarder?
Every Way I suggest to ask to Splunk Support.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...