Getting Data In

Turn THP off on Universal Forwarder?

mfrost8
Builder

I get the whole thing about turning off THP on Splunk Enterprise instances per https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP and many other places.

However, everything I've seen refers pretty specifically to Splunk Enterprise instances. I would have assumed that this wouldn't be necessary on a universal forwarder, yet on our late-model UF installations, I always see:

04-24-2017 07:33:50.452 -0500 WARN  ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.

Being, the same message as you'd get on a Splunk Enterprise instance, it seems that Splunk wants this turned off on universal forwarder installations too? I'm somewhat less comfortable with that on servers whose function is exclusively Splunk as the impact of THP is unknown to me.

Or perhaps is the existence of this warning on a universal forwarder a bug of some sort? I'm pretty sure I've seen it there for quite a few versions now.

Thanks!

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi mfrost8,
I didn't experienced problems related to THP on forwarders because "On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance", see (https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP).
In addition in http://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements they speak about Splunk Enterprise and not about Splunk Forwarders.

Bye.
Giuseppe

View solution in original post

0 Karma

mfrost8
Builder

I opened a case with Splunk on this. Despite the same dire warning about THP, it's probably not quite as essential on a universal forwarder. One would need to look at the specifics of THP and the generalizations about how it impacts applications.

It seems that Splunk could benefit from it when you've got a lot of open files. Perhaps less essential if you've only got a few. It's not likely to be impactful to a non-Splunk app, unless it's doing operations on large files on a system with very little memory.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi mfrost8,
I didn't experienced problems related to THP on forwarders because "On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance", see (https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP).
In addition in http://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements they speak about Splunk Enterprise and not about Splunk Forwarders.

Bye.
Giuseppe

0 Karma

mfrost8
Builder

Right, but then why the warning in the universal forwarder logs?

0 Karma

gcusello
SplunkTrust
SplunkTrust

I have many Red hat Forwarders but I haven't this message.
Are you using Universal or Heavy Forwarder?
Every Way I suggest to ask to Splunk Support.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...