When THP is enabled in our environment, we notice immediately (we are not in charge of our servers and this happens occasionally with OS patches etc). A few things that we watch/notice:
1. Our environment is very heavy on searches. If we timechart the CPU of all of our indexers in a line graph, they follow a very nice pattern - spikes on the 0, 15, 30, 45 minutes due to more searches running on those minutes. The lines on the graph are close together and look almost like someone has used a rainbow pen to draw a single line. When THP is enabled, there is variation to this and the graph just doesn't look the same - the lines for each indexer are distinct and don't necessarily follow the same pattern. On average, CPU is a bit higher, but it is really the timechart of the CPU graph that is telling.
2. We have some scheduled searches that run every minute or every 5 minutes. We timechart the avg() or perc90() of the run time of those searches and when THP is on, that will be higher by a noticeable amount. The timechart of the avg() allows us to account for the variations in load.
You can check for THP by looking at the internal logs and the ulimit parameter. Splunk only does this when it is starting up, so you will need to look at the ulimit after the THP has been turned off and splunk has restarted.
... View more