Configure a Vsphere VM for Splunk

[ironaddict]

Hello,

How do I configure a vSphere VM (Windows Server 2016) for a SPLUNK deployment?

So far I have done the following:

• 12 vCPU
• 12 GB of RAM
• Full reservations for vCPU and vRAM (No CPU and memory overcommit)
• Use VMware Tools in the guest VM
• Use VMXNET3 network adapter

How do I check or configure these settings?

• Raw volumes may provide slight improvements over VMFS
• Minimum 1200 random seek operations per second disk performance (sustained)
• Use VMware Paravirtual SCSI (PVSCSI) controller
• Provision virtual disks as Eager Zero Thick when not on an array that supports the appropriate VAAI primitives (“Write Same” and “ATS”)
• NFS and iSCSI disks are not recommended due to higher latency and file locking issues.

[Richfez]

For the most part, these things may affect performance, but won't generally make it not work, so you should be OK. In fact, the simple fact of asking and being worried about them implies you'll pay attention later and if you start having problems, you'll come back to this list and double-check some things. (The one that could be very problematic is the IOPS - point 2.)

Almost in order:

The recommendation that a raw volume may perform better than VMFS is certainly true, mostly, but it's not important. The difference nowadays is practically nothing, and the drawbacks are massive (no vmotion or snapshots or ... or ...). If you have a dedicated storage team that works really well with your dedicated VMware team, then sure - go all out. If you, like most medium sized and smaller companies do not have individual teams and tens of millions of dollars of SANs and VMware infrastructure, don't. Just use VMFS. You'll thank me later.

1200 IOPS can be confirmed with something like bonnie++. Shared storage is nearly always a problem - it really takes a moderately beefy and not overloaded SAN to do more than 1200 IOPS for your server all the time. Yeah, that SAN can do 25k IOPS. But it also is running 300 other systems that all require some IOPS, so it's shared with all. And then OH BACKUPS OMG, where did my IOPS go? I mean, your Splunk server may be ingesting more data during backups than at any other time, depending on what you are ingesting, so having your IOPS drop to 300 right when you need them most is a problem. BUT - don't get too worked up over this right now. Best to confirm you at least can run a reasonable number of IOPS usually and then know that this will be your problem later if you have any problems with performance. E.g. don't blame Splunk when that happens. 🙂

I'm jumping to the last thing next - NFS and iSCSI aren't always the greatest. Splunk has specific issues especially with NFS, as they mention. You probably aren't using either if you don't recognize 'NFS' or 'iSCSI', because if you were paying attention when you built the datastores or the VMFS volumes, I believe both show up in the ... well, in something ... when you build those. But either way, this is a VMware question, really - here's a start at explaining (with an old version of ESX) about the two and about storage in general. Try this link to vmware's docs. (And I'd suggest perhaps using keywords from in there to search for a version that's more like your own version of VMware - 5.1 is rather old now).

So, the 3rd and 4th question.
The VMware paravirtual SCSI controller - just google it. It's a thing. VMware has docs for it. Why to use it, how to use it, what you have to watch for or make sure happens before you use it (mostly OS-related), etc...

Same thing with Eager Zero Thick. Use the search, Luke! lol. Anyway, you can thick provision or thin provision VMFS volumes. Thin provisioned means you don't actually allocate real actual disk space on the SAN/Device when you create the disk but instead VMware requests new "space" as it needs it. So if you give it a 100 GB thin provisioned disk, it only actually "creates" a disk that's as big as your OS needs, let's say 10 Gb. Then as you use the system, it writes to 'disk' and it needs more, so as it needs more VMware asks for it to ... have more. (We'll come back to that). If you thick provision, it allocates that whole pile o' disk all up front, so it's already there. Now, in thin provisioning, requesting more disk does take some time so adding data to a disk takes longer than it should, and the disk gets really fragmented sometimes (which isn't like the old Windows fragmentation - this is something of an issue, but usually has to be really bad to be a real problem, depending on your SAN). What they're saying in the second part of this is that if the SAN supports VAAI primitives, it can do these extensions super-fast, like they're not even slowing things up so this is then fine. So if your San supports that, you can thin provision, if not, thick provision. Now, how do you find out if yours supports VAAI? Google again will help you there - you'll find VMWare docs that tells you exactly how you tell if yours does, how to enable it if it does and it isn't enabled, and so on. So, again, google or even Bing will help you there.

I hope this helps!

Happy Splunking,
Rich