Getting Data In

Turn THP off on Universal Forwarder?

mfrost8
Builder

I get the whole thing about turning off THP on Splunk Enterprise instances per https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP and many other places.

However, everything I've seen refers pretty specifically to Splunk Enterprise instances. I would have assumed that this wouldn't be necessary on a universal forwarder, yet on our late-model UF installations, I always see:

04-24-2017 07:33:50.452 -0500 WARN  ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.

Being, the same message as you'd get on a Splunk Enterprise instance, it seems that Splunk wants this turned off on universal forwarder installations too? I'm somewhat less comfortable with that on servers whose function is exclusively Splunk as the impact of THP is unknown to me.

Or perhaps is the existence of this warning on a universal forwarder a bug of some sort? I'm pretty sure I've seen it there for quite a few versions now.

Thanks!

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi mfrost8,
I didn't experienced problems related to THP on forwarders because "On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance", see (https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP).
In addition in http://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements they speak about Splunk Enterprise and not about Splunk Forwarders.

Bye.
Giuseppe

View solution in original post

0 Karma

mfrost8
Builder

I opened a case with Splunk on this. Despite the same dire warning about THP, it's probably not quite as essential on a universal forwarder. One would need to look at the specifics of THP and the generalizations about how it impacts applications.

It seems that Splunk could benefit from it when you've got a lot of open files. Perhaps less essential if you've only got a few. It's not likely to be impactful to a non-Splunk app, unless it's doing operations on large files on a system with very little memory.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi mfrost8,
I didn't experienced problems related to THP on forwarders because "On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance", see (https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP).
In addition in http://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements they speak about Splunk Enterprise and not about Splunk Forwarders.

Bye.
Giuseppe

0 Karma

mfrost8
Builder

Right, but then why the warning in the universal forwarder logs?

0 Karma

gcusello
SplunkTrust
SplunkTrust

I have many Red hat Forwarders but I haven't this message.
Are you using Universal or Heavy Forwarder?
Every Way I suggest to ask to Splunk Support.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...