Has anybody implemented a distributed Splunk Environment using Virtual Machines from top to bottom?
This seems to be hardly an issue for most of the components, but the Indexer seems to generate the most worry. Are there any resources outlining any unique configuration that would make a virtual Indexer perform better?
Most critical aspects of a successful VM deployment:
- vCPU reservation of at least 8 cores, 12 is better
- vRAM reservation of at least 12GB
- eager-zero provisioned disk providing at least 800IOPS concurrently per indexer
As long as you don't oversubscribe and configure following the recommendations in the TechBrief, an indexer can work well in a virtual environment. Did I mention resource reservations?
If your VM hosts are oversubscribed, you don't reserve resources and you are ending up with high CPU Ready counts; or if your underlying disk is not performing at the recommended rates, things will probably not live up to your expectations.
Editor's note: 2014 below. 2016 version here: https://www.splunk.com/pdfs/technical-briefs/splunk-deploying-vmware-tech-brief.pdf
Check out this URL..
https://www.splunk.com/web_assets/pdfs/secure/Splunk_and_VMware_VMs_Tech_Brief.pdf
We started off with our indexers as VM's. It worked OK in the beginning, as we moved on it turned into a bottle neck. So we ended up buying hardware for the indexers. Our search heads run fine on a VM.
Editor's note: 2014 below. 2016 version here: https://www.splunk.com/pdfs/technical-briefs/splunk-deploying-vmware-tech-brief.pdf
Check out the following Splunk whitepaper:
http://www.splunk.com/web_assets/pdfs/secure/Splunk_and_VMware_VMs_Tech_Brief.pdf