When installing UF, should THP be invalidated or not?

[yutaka1005]

It is related to the following answers, but is it recommended to invalidate THP after all?

https://answers.splunk.com/answers/523835/turn-thp-off-on-universal-forwarder.html

If it does not seem to affect indexing and search performance only,
It seems there is no need to invalidate it.
Also, if it is recommended, I would like to know how it is affected.

I hope someone can tell me.

[yutaka1005 · Accepted solution]

Splunk support said it recommends disabling THP.

[mayurr98]

On systems with THP enabled, Splunk has observed a minimum of a 30% degradation in indexing and search performance, with a similar percentage increase in latency.

here is a blog on how it affects indexing performance
https://blogs.oracle.com/linux/performance-issues-with-transparent-huge-pages-thp

In this official doc, they have beautifully described why THP must be disabled.
https://docs.splunk.com/Documentation/Splunk/7.0.1/ReleaseNotes/SplunkandTHP

Right now in your environment, you are not facing performance issues because of hardware capacity and data might be less. But eventually, when it grows you will tend to face performance issues. So it is better to disable it now only.

Let me know if this helps !

[nickhills]

Since the Splunk code base is essentially the same for Enterprise functions (search/index/etc) and the universal forwarder, it is reasonable to assume that any impact kernel settings have affect the the UF application as well as other components, however, the workload executed by a UF is significantly different to that performed by indexers and search heads.

THP has the potential to affect Splunk in 3 ways:
- The implementation is too aggressive at coalescing memory pages for short-lived processes (such as many Splunk searches)
- It can prevent the jemalloc memory allocation implementation from releasing memory back to the operating system after use. [snip].
- For some workloads, it can cause I/O regressions surrounding swapping of huge pages.
https://docs.splunk.com/Documentation/Splunk/7.0.1/ReleaseNotes/SplunkandTHP

As you might guess, these impacts are centered around high memory use operations with regard to splunk specific workloads - which not a work profile a UF displays.

I have done extensive testing of THP on indexers, searcheads and UFs and whilst I have seen performance differences in both search and index tiers, I have not seen any change for universal forwarders.

In my environments THP is therefore only disabled for enterprise servers, and left enabled for UFs.

[yutaka1005]

Thank you for answer!

I heard Splunk support about this, and they said it recommends disabling THP.

However, in your verification, disabling of THP had no influence to UF.
So perhaps, you may not need to disable THP in environments where THP has to be activated.