Getting Data In

When installing UF, should THP be invalidated or not?

yutaka1005
Builder

It is related to the following answers, but is it recommended to invalidate THP after all?

https://answers.splunk.com/answers/523835/turn-thp-off-on-universal-forwarder.html

If it does not seem to affect indexing and search performance only,
It seems there is no need to invalidate it.
Also, if it is recommended, I would like to know how it is affected.

I hope someone can tell me.

0 Karma
1 Solution

yutaka1005
Builder

Splunk support said it recommends disabling THP.

View solution in original post

0 Karma

yutaka1005
Builder

Splunk support said it recommends disabling THP.

0 Karma

mayurr98
Super Champion

On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance, with a similar percentage increase in latency.

here is a blog on how it affects indexing performance
https://blogs.oracle.com/linux/performance-issues-with-transparent-huge-pages-thp

In this official doc, they have beautifully described why THP must be disabled.
https://docs.splunk.com/Documentation/Splunk/7.0.1/ReleaseNotes/SplunkandTHP

Right now in your environment, you are not facing performance issues because of hardware capacity and data might be less. But eventually, when it grows you will tend to face performance issues. So it is better to disable it now only.

Let me know if this helps !

0 Karma

nickhills
Ultra Champion

Since the Splunk code base is essentially the same for Enterprise functions (search/index/etc) and the universal forwarder, it is reasonable to assume that any impact kernel settings have affect the the UF application as well as other components, however, the workload executed by a UF is significantly different to that performed by indexers and search heads.

THP has the potential to affect Splunk in 3 ways:
- The implementation is too aggressive at coalescing memory pages for short-lived processes (such as many Splunk searches)
- It can prevent the jemalloc memory allocation implementation from releasing memory back to the operating system after use. [snip].
- For some workloads, it can cause I/O regressions surrounding swapping of huge pages.
https://docs.splunk.com/Documentation/Splunk/7.0.1/ReleaseNotes/SplunkandTHP

As you might guess, these impacts are centered around high memory use operations with regard to splunk specific workloads - which not a work profile a UF displays.

I have done extensive testing of THP on indexers, searcheads and UFs and whilst I have seen performance differences in both search and index tiers, I have not seen any change for universal forwarders.

In my environments THP is therefore only disabled for enterprise servers, and left enabled for UFs.

If my comment helps, please give it a thumbs up!

yutaka1005
Builder

Thank you for answer!

I heard Splunk support about this, and they said it recommends disabling THP.

However, in your verification, disabling of THP had no influence to UF.
So perhaps, you may not need to disable THP in environments where THP has to be activated.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...