topic Re: Splunk indexers IOPS tade off in Getting Data In

Splunk indexers IOPS tade off

egsub — Tue, 31 Mar 2015 13:27:51 GMT

Hi,

We're considering to raise the number of indexers servers (peers) with reducing the IOPS per server FS due to IOPS limitations in our hardware.

Does this calculation fit? For instance, does 10 servers with 600 IOPS S-ATA based are equal to 5 servers with 1200 IOPS SAS based in case of 500GB/day indexed data?

Thanks,
EG

Re: Splunk indexers IOPS tade off

jmheaton — Tue, 31 Mar 2015 21:22:54 GMT

You should be more than fine with going with 5 servers at 1200 SAS.
A good rule of thumb is to go 1 server, 8 cores, 8 GB ram, and 1000 IOPS per 100GB daily data.
Do you actually index all 500GB/day

Re: Splunk indexers IOPS tade off

ekost — Wed, 01 Apr 2015 00:44:55 GMT

There's no magic answer available, as the type of search and the data volume per day will have a large impact on relative performance. If your SATA-based storage delivers the promised 600 IOPS (test with bonnie++,) and the type of searches you are running are balanced between CPU and I/O bound, and the data volume per-day per-indexer averages ~100GB/Day, and the data coming from the forwarders is evenly distributed across all indexers, and assuming an identical CPU core count, then having 2x indexers with less IOPS should equal your other hardware. There is no substitute for testing with actual data and a search and indexing load. Please note that a lot of admins discover that poor search result speed causes user aggravation, and insufficient IOPS can be a factor.

Re: Splunk indexers IOPS tade off

esix_splunk — Wed, 01 Apr 2015 04:49:00 GMT

It should also be noted that 900iops is the minimum recommend spec for Splunk 6.2+. If you arent able to meet these requirements and decide to deploy lower spec hardware, you need to be careful and carefully monitor the Splunk Queues. Most likely you're going to see issues and need to modify configurations on your outputs to the indexers etc.