Infrastructure recommendations

[annebeate]

Hi,

We are planning to upgrade our Splunk environment to be able to handle increased load. We currently have four physical Splunk servers, IBM System x3650 M3 – 3,46GHz, in our environment using local disks. 2 index servers and 2 search heads running on Redhat Linux 64 bits. Index servers have physical disks 10x300Gb RAID10 Stripe-based on /opt. RAM: 24GB (1 search head and 1 index server) and 16GB memory (1 search head and 1 index server). 2xCPU (4 core) for index server. 2xCPU (6 core) for search head.

We are thinking about exchanging these servers with

• 3 VMware servers for search heads using Redhat Linux and SAN disk. 16 GB RAM. 4 CPU. 3GHz. 1 of the search heads should handle saved searches used for generating alerts.

• 4 VMware servers for index servers using Redhat Linux and SAN disk. Minimum 800 iops. 16 GB RAM. 4 CPU. 3GHz.

• 2 VMware servers for intermediate forwarders between servers sending logs to Splunk and the index servers. 4GB RAM, 1 CPU. 3GHz.

We currently have about 150 forwarders. Daily index usage is on average between 30-40GB.
We have approx 400 users defined. On of the search heads is used a lot for generating alerts.

Could you please make some recommendations regarding our infrastructure strategy? We are planning to use the latest version of Splunk in the new environment.

Best regards,
Anne

[lguinn2]

Instead of increasing your resources, it looks like your virtual environment will have fewer resources. I think you will be disappointed with the results if you use the configuration that is proposed. Here are a few ideas and resources.

From the Splunk Installation manual:

"If you run Splunk in a virtual machine (VM) on any platform, performance does degrade. This is because virtualization works by abstracting the hardware on a system into resource pools from which VMs defined on the system draw as needed. Splunk needs sustained access to a number of resources, particularly disk I/O, for indexing operations. Running Splunk in a VM or alongside other VMs can cause reduced indexing performance."

See also Hardware capacity planning and this technical brief on using Splunk with VMware

My suggestions:

Search heads are often CPU bound. Recommend 16 CPUs per search head. Set a CPU reservation. 32 MB memory might be helpful, but the CPUs are much more important.

Indexers: 800 IOPS would be fine for a small environment. Recommend 1200 IOPS for this environment. Also, at least 8 CPUs per indexer to support the search load. Again, set a CPU reservation. You should use local disks if at all possible, as shared IO is a performance bottleneck in many virtual environments. Raw device mapping (RDM) may also improve performance.

Final suggestion: test under load. Sometimes these VMs look fine in a test environment, but become sluggish immediately when they have to share resources with production VMs.

Caveat: my VCP 4 is pretty stale at this point.

[annebeate]

Thanks a lot for valuable feedback, I will look into the links you referred to.

Thanks, Anne