Archive

How does Splunk handle Hyperthreading in a VMware environment?

slea_splunk
Splunk Employee
Splunk Employee

This document is helpful for architecting Splunk in a VMware environment. However, is there any guidance on how Splunk handles Hyperthreading in this environment?

Tags (1)
0 Karma

hsesterhenn_spl
Splunk Employee
Splunk Employee

Hi,

as mentioned in the TechBrief there are some things to take care of if you run Splunk Enterprise at scale.
Rich7177's answers is correct for really small environments.

Regarding HT, Splunk is able to detect this. Look for log lines like "Detected 32 (virtual) CPUs, 16 CPU core" in you splunkd.log.

VMware e.g. will count every "virtual" core and at the end the best practices approach is to use 2x number of vCores compared to a bare metal setup.

Meaning, instead of 12 cores for the reference architecture you'll configure 24 "vCores" for your VMware instance.

HTH,

Holger

jaracan
Communicator

Hi @hsesterhenn_spl ,

You mention that Splunk is able to detect this Hyperthreading by looking for log lines like "Detected 32 (virtual) CPUs, 16 CPU core" in splunkd.log. Does this only apply for Virtual Machines? Do this event also be seen if the server is a Physical Server?

Regards,
jaracan

0 Karma

hsesterhenn_spl
Splunk Employee
Splunk Employee

Hi @jaracan,

the "loader" message in splunkd.log at startup or the Monitoring Console are showing the same information.

http://<SPLUNK-IP>:8000/en-GB/app/splunk_monitoring_console/resource_usage_machine

It will just report what the OS is presenting. Nothing special regarding the underlying virtual solution.

If you have CPU=vCPU then most of the time you can be sure it's running in any kind of virtual environment because most of the servers are running with active HT nowadays. 

But it depends on the virtual solution how they translate the HT "cores/threads" into the information which can be read from the OS.

So, I would call it "educated" guess if you see CPU=vCPU.

Does it make sense?

Greetings,

Holger

 

0 Karma

jaracan
Communicator

Hi @hsesterhenn_spl ,

I ran "lscpu" on one of the Indexers (Physical Server). Since Threads per core is one (1) but on On Flags, we can see "ht". Is the hyperthreading really enabled or not? Quite confused since threads per core is not 2.

jaracan_0-1605866640568.png

 

 

 

0 Karma

hsesterhenn_spl
Splunk Employee
Splunk Employee

Hi, @jaracan,

It's been a while since my deep VMware and Linux times...

But IIRC, the lscpu will show the attribute bits which are visible, and because of "VT-x" you see "HT".

But still only one thread per CPU.... maybe because of VMware doing things differently?

No matter what... the information shown in Splunk is the important one for us. 

And you can open a full can of worms using lscpu...

https://unix.stackexchange.com/questions/33450/checking-if-hyperthreading-is-enabled-or-not#comment7...

Greetings,

Holger

 

0 Karma

Richfez
SplunkTrust
SplunkTrust

There's a longer answer that probably requires expertise that isn't quite as out of date as mine, but generally Splunk and Hyperthreading get along fine, regardless if it's a physical or virtual environment. They are not full cores, I'd not pretend they were nor hope they'll add nearly as much increased performance, but they should provide moderate performance increases.

VMware IIRC has HT enabled by default if it finds a HT enabled host. It handles scheduling in a slightly different way than when there is no HT enabled, but in the end it shouldn't really matter - assign resources as appropriate for your loads, as based on how they're performing and how much CPU/RAM they're using, adjust as necessary, keeping an eye on your hosts for how heavily you are loading them and so on. Essentially, run your VMware infrastructure and Splunk infrastructure just like you should - by monitoring it and keeping track of what it's doing, incrementally expanding either as required to keep pace with needs.

Also note that Splunk works fine with quite a few CPUs, so even if an "HT enabled 12 cores" isn't keeping up, making it 16, 20, 24, or even more is fine. Obviously assuming you have that many on an individual host, and keeping in mind the effect of NUMA boundaries.

Hope this helps!
-Rich

0 Karma

sloshburch
Ultra Champion

Comment from @tmuth_splunk:

The biggest I see with hyperthreading is the misunderstanding of how to count cores vs threads. If hyperthreading is enabled (which it is most of the time), a vCPU is a thread, so allocating 8 vCPUs is actually 4 cores.

Comment from @sdvorak_splunk:

The VMware scheduler will try to allocate physical cores before using hyper-threaded cores unless it detects an operation that is suitable for hyper-threaded core(s).
It is somewhat dependent on the hypervisor version. Older versions weren’t so good at this. The VMware administrators should be able to make a recommendation. That said, check with VMware, but currently, hypervisor versions do it this way.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!