Getting Data In

Turn THP off on Universal Forwarder?

mfrost8
Builder

I get the whole thing about turning off THP on Splunk Enterprise instances per https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP and many other places.

However, everything I've seen refers pretty specifically to Splunk Enterprise instances. I would have assumed that this wouldn't be necessary on a universal forwarder, yet on our late-model UF installations, I always see:

04-24-2017 07:33:50.452 -0500 WARN  ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.

Being, the same message as you'd get on a Splunk Enterprise instance, it seems that Splunk wants this turned off on universal forwarder installations too? I'm somewhat less comfortable with that on servers whose function is exclusively Splunk as the impact of THP is unknown to me.

Or perhaps is the existence of this warning on a universal forwarder a bug of some sort? I'm pretty sure I've seen it there for quite a few versions now.

Thanks!

0 Karma
1 Solution

gcusello
Legend

Hi mfrost8,
I didn't experienced problems related to THP on forwarders because "On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance", see (https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP).
In addition in http://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements they speak about Splunk Enterprise and not about Splunk Forwarders.

Bye.
Giuseppe

View solution in original post

0 Karma

mfrost8
Builder

I opened a case with Splunk on this. Despite the same dire warning about THP, it's probably not quite as essential on a universal forwarder. One would need to look at the specifics of THP and the generalizations about how it impacts applications.

It seems that Splunk could benefit from it when you've got a lot of open files. Perhaps less essential if you've only got a few. It's not likely to be impactful to a non-Splunk app, unless it's doing operations on large files on a system with very little memory.

0 Karma

gcusello
Legend

Hi mfrost8,
I didn't experienced problems related to THP on forwarders because "On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance", see (https://docs.splunk.com/Documentation/Splunk/6.5.3/ReleaseNotes/SplunkandTHP).
In addition in http://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/Systemrequirements they speak about Splunk Enterprise and not about Splunk Forwarders.

Bye.
Giuseppe

0 Karma

mfrost8
Builder

Right, but then why the warning in the universal forwarder logs?

0 Karma

gcusello
Legend

I have many Red hat Forwarders but I haven't this message.
Are you using Universal or Heavy Forwarder?
Every Way I suggest to ask to Splunk Support.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...