Splunk and Hyper-threading - Has anyone run benchmarks?

[gfriedmann]

I'm wondering weather using hyper-threading would be a good idea or not. I am hoping somebody has tried this with 4.x. I'm mostly interested in search performance, not indexing.

sometimes typed as "hyperthreading"

[gfriedmann · Accepted solution]

Quick Result Summary

You probably want to enable hyper-threading and disable power saving bios configs. The hyper-threading benefits can be notable but do not kick in until you have more search jobs than physical cores. But if the server is usually under utilized (in terms of jobs/physical core), then hyperthreading could hurt more than help if power saving options are enabled in the system BIOS.

Testing

With the help of the perftest application, i have gone through multiple rounds of performance testing.

My test was 1 - 30 parallel CPU bound search jobs over the same sample data. The search individually could complete in about 60 seconds.

The hardware is Intel Xeon X5680 @ 3.33GHz (Dual 6-core server)

I tested the following cases

A) Disable hyper-threading. Enable Power Saving BIOS Config
B) Enable hyper-threading. Enable Power Saving BIOS Config
C) Enable hyper-threading. Disable Power Saving BIOS Config
D) Disable hyper-threading. Disable Power Saving BIOS Config 

Test Results

Median search completion time by case by # of parallel searches.

                      CASE
               A     B     C     D
1 search      70    82    58    59
2 search      62   100    59    58
8 search      63    75    63    63
16 search     97    85    86    98
20 search    121    99    99   122 
30 search    180   144   144   182

Notable Observations

• A single search load was most volatile. I suspect this is due to power throttling for partially used processor resources.
• Hyper-threading on with power savings can increase search time by ~40% when load is low (less searches than physical cores). (see case B)

---

Definition

power savings BIOS config - Settings such as:

• HP power Profile
• HP Power Regulator
• Turbo boost Optimization Dynamic
• Power Savings Mode Response Minimum
• Processor Idle Power States (C-States)

[jodros]

To further support this answer, micron just finished very similar benchmarking, noting performance impacts from BIOS changes and how that affects IO. Other BIOS setting tweaks are mentioned that might also yield gains. As we utilize SSD's in all 4 of our indexers, CPU is our current bottleneck and any sort of tuning would be a positive impact.

http://www.micron.com/~/media/Documents/Products/White%20Paper/pcie_ssd_performance_hi.pdf

[LCM]

This question could be similarish to the question I had recently. Since you are looking for the search performance & hyperthreading, I reckon hyperthreading is no option for that:

http://answers.splunk.com/questions/12027/single-multi-threading-cpu