Officially, Splunk recommends that if you're going to virtualize, that you should use a many virtual cores/CPUs as for physical, and furthermore to avoid the problem you note (having to wait for 8 CPUs to come free before allowing the VM to run anything) that the CPU resources be dedicated to the VM, i.e., they are always available because they can't be used for anything else. I like say that you shouldn't be looking to save hardware via virtualizing Splunk, only to gain flexibility.
This is because we find that Splunk actually does wind up using and needing the physical CPU resources. Now, it is possible that right now you are at a place where you only need 4 cores for your workload, and in that case you may find that you run into less CPU wait if you reduce the number of cores. But it should be noted that when Splunk uses the CPUS, it will actually use the CPUs and you need to have the underlying physical CPUs available to perform the work. People like to get all crunky about the waiting for free CPUs in VMware, but you may find that you actually need the physical resources to do the search work, so be prepared to bump it back up. (Also, I suspect that you may not be seeing problems because in fact the CPUs are busy enough that it never lets go of its 8 CPUs.)
As for the messages you are seeing about "maximum concurrent searches", first, you should be sure that the messages are not coming from user role restrictions, but are in fact about the global system maximum. if that is the case, and you really aren't seeing problems, you should feel free to increase either the base or multiplier for the number of searches per CPU. The default in 5.x is 2+2xcores, but you can up to to 3x or possibly even 4x cores. It is possible depending on your search profile that you can run more of them, though a different workload may cause searches to backlog.
... View more
