Search Head VM Sizing Help

[mfrost8]

We are looking to build out a search head for the first time and I'm a little confused as to how it should be sized. Our Splunk usage is growing practically by the day and it's many heterogenous apps and teams using it. It's hard to get an exact handle on what's being done today let alone what they're going to want to do tomorrow. Our org twists our arms mightily to do use VMs (we are using physicals for indexers, of course). Our license volume is currently around 10-15GB/day, but we just moved to a 50GB/day license and are expanding to new applications.

I've read through

http://docs.splunk.com/Documentation/Splunk/latest/installation/capacityplanningforalargersplunkdepl...

and

http://splunk-base.splunk.com/answers/298/can-i-run-splunk-in-a-vm-are-there-any-issues-or-tricks-i-...

I was thinking 8 CPUs in a VM (probably overkill for today, but hopefully not in the long run). However, the amount of memory I should request and in particular the amount of disk I'd want to add is a mystery. The first doc seems to imply I'd want 300GB of usable space on a search head. I realize it all depends on the kinds of searches people would run, but is 300GB a little excessive for disk space on a search head?

What about memory? If the search head is more CPU intensive, it's probably going to use a reasonable chunk of memory aggregating that data, doing search-time field extractions, rendering dashboards, etc, right?

I'm curious what people who run search heads as VMs have sized them to and what their experiences are.

Thanks!

[pmadnyl]

lguinn, thanks for the info. i'm new to splunk. and we are looking to do a similar thing, have search heads on vm windows, but indexers on linux servers. the reference hardware recommendation looks to be for physical boxes. what is the recommendation for vm windows or vm linux for search heads?

[lguinn2 · Accepted solution]

The documentation recommends

• Intel 64-bit chip architecture
  
• Standard Linux or Windows 64-bit OS
• 4 CPUs, 4 core per CPU (2,5 - 3 GHz)
• ≈4GB RAM
• 2 300GB SAS hard disks at 10,000 rpm each in RAID 0
  
• standard 1Gb Ethernet NIC, optional 2nd NIC for a management network
  

for a search head. In my experience, this is a pretty good place to start. This will allow over a dozen simultaneous searches. You could probably reduce the disk space; most of what you will be storing will be configuration files. But be aware that summary indexes are created on the search head by default. So either make sure that you have enough disk if using summary indexing, or configure the summary indexes to be created on the indexers instead.

The bottleneck for search heads is usually CPU, so more is better. Remember that each search uses a separate CPU, so the number of CPUs determines the maximum number of simultaneous searches. More memory is not usually helpful. But in a VM, it is easy to increase memory if needed.

Of course, if searches are slow, a good way to improve performance is to add more indexers - indexers do the work of actually searching!

[pmadnyl]

guinn, thanks for the info. i'm new to splunk. and we are looking to do a similar thing, have search heads on vm windows, but indexers on linux servers. the reference hardware recommendation looks to be for physical boxes. what is the recommendation for vm windows or vm linux for search heads?

[lguinn2]

Well, you could certainly have multiple search heads. So if you really want to make a search head as a VM, maybe you could give it 8 vCPUs and then just have twice as many search heads!

[mfrost8]

Ooof. This seems like a harder box to be a VM (at least in our environment). I think I had interpreted that the docs meant these specs were for a box that was a sort of combined search head and indexer before one split those two out. Apparently not.

Thinking about it, I would think that the most disk space would probably be used by the dispatch dir.

I will need to re-examine whether we do this as virtual or physical now. Thanks for the help, Iguinn.