Turn THP off on Universal Forwarder?

[mfrost8]

I get the whole thing about turning off THP on Splunk Enterprise instances per https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP and many other places.

However, everything I've seen refers pretty specifically to Splunk Enterprise instances. I would have assumed that this wouldn't be necessary on a universal forwarder, yet on our late-model UF installations, I always see:

04-24-2017 07:33:50.452 -0500 WARN  ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.

Being, the same message as you'd get on a Splunk Enterprise instance, it seems that Splunk wants this turned off on universal forwarder installations too? I'm somewhat less comfortable with that on servers whose function is exclusively Splunk as the impact of THP is unknown to me.

Or perhaps is the existence of this warning on a universal forwarder a bug of some sort? I'm pretty sure I've seen it there for quite a few versions now.

Thanks!

[mfrost8]

I opened a case with Splunk on this. Despite the same dire warning about THP, it's probably not quite as essential on a universal forwarder. One would need to look at the specifics of THP and the generalizations about how it impacts applications.

It seems that Splunk could benefit from it when you've got a lot of open files. Perhaps less essential if you've only got a few. It's not likely to be impactful to a non-Splunk app, unless it's doing operations on large files on a system with very little memory.

[gcusello · Accepted solution]

Hi mfrost8,
I didn't experienced problems related to THP on forwarders because "On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance", see (https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP).
In addition in http://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements they speak about Splunk Enterprise and not about Splunk Forwarders.

Bye.
Giuseppe

[mfrost8]

Right, but then why the warning in the universal forwarder logs?

[gcusello]

I have many Red hat Forwarders but I haven't this message.
Are you using Universal or Heavy Forwarder?
Every Way I suggest to ask to Splunk Support.
Bye.
Giuseppe