Splunk Search

The percentage of Read / Write utilization to get to 800 IOPS?

maverick
Splunk Employee
Splunk Employee

I am trying to figure out the drive configuration to meet the recommended 800 IOPS noted in the Splunk documentation

What is the percentage of Read vs Write utilization needed to meet 800 IOPS?

1 Solution

dwaddle
SplunkTrust
SplunkTrust

Sorry, but the best answer is "it depends on your workload." Read IOPS depends greatly on search activity. Write IOPS depends on indexing throughput. With the exception of caching, the disk subsystem doesn't care whether the 800 IOPS is read or write. A good cache can obviously greatly improve both read and write IOPS. (This includes operating system cache and controller cache).

Borrowing from http://www.symantec.com/connect/articles/getting-hang-iops (a most excellent article BTW), single drive IOPS is strongly related to avg drive seek time. Modern 15K RPM SAS drives can range from 175 to 210 IOPS per drive. Striping multiples IOPS, mirroing divides them.

If you can get ahold of 210 IOPS drives (which would have the lowest average seek time), then 4 of those in a RAID0 stripe gives you 840 IOPS (without cache). Put 8 of them into a RAID10 (4 striped mirrored against 4 more striped) is a solid 800IOPS solution. The lower-end 175 IOPS 15K RPM SAS drives in the same configuration could give 700 IOPS.

One substantial issue here is that most name-brand server vendors don't give you enough information behind a specific drive model. They may have two or three different drives they supply under the same generic "part number", each with different specs. (We have a big SAN array that uses Seagate and Maxtor drives side-by-side)

View solution in original post

kiril123
Path Finder

The percentage of Read vs Write utilization should come from your specific Splunk usage scenarios.
Are you running a lot of searches/reports but the indexing volume is less compared to that?

Lets say you think that 40% will be write utilisation and 60% read utilisation.

The formula to calculate IOPS is as the following:

IOPS_total = IOPS_single_drive*number_of_drives/(R + F*W)

IOPS_single_drive - you can obtain this from disk manufacturer or calculate by dividing 1 by disk latency in ms.
number_of_drives - number of drives you are planning to deploy.
R - read utilisation. In our case it is 60% as defined above or 0.6.
W - write utilisation. In our case it is 40% as defined above or 0.4.
F - RAID penalty factor. It depends on the RAID type. For RAID1 as an example for every write request to the controller there are 2 writes to the disks. So the penalty factor is 2. For RAID5 for every write request to the controller there are 4 writes to the disks. Penalty factor is 4 in this case.

0 Karma

anwarmian
Communicator

My two cents worth: SSD is the way to go. It does provide much higher IOPS than SAS, SATA HDD or Hybrid. I'm talking from my own experience. If you have RAID 10 with SSD's you should be set.

0 Karma

gregschulz
Explorer

Btw, I mentioned HHDDs (Hybrid Hard Disk Drives) however I did not say what they are (my bad).

Here is a link (

2011 Summer momentus hybrid hard disk drive (HHDD) moment
http://storageioblog.com/?p=2075 )

to a series of posts that I have been doing covering how I have been using HHDDs for over a year to boost performance in my workstations and laptops. In simple terms, solid state devices (SSD) are very fast providing many times more IOPS (or bandwidth) than HDDs, however at a higher cost. HDDs provide a large amount of space capacity and lower number of IOPs compared to SSDs for a given price.

HHDDs provide best of both worlds in that they are externally physically identical to a HDD and depending on implementation will be plug compatible with your controller/adapter transparent to your server or workstation or laptop operating system.

There should be no need for special drivers, adapters, migration or management software such as is the case with tiering or automated movement technologies. The benefit with HHDDs is that you get some performance boost in that there is a small SSD integrated with the internal HDD processor to function as a super persistent (no data loss when power turned off) cache for the underlying HDD.

Im using the Seagate Momentus XTs which are 7.2K RPM 500GB 2.5" form factor SATA HDDs that have an integrated 4GB SLC flash SSD plus 32MB DRAM buffer. These HHDDs appear to windows or apple OS as a regular disk with no special drivers yet enable more IOPS than a HDD. If the data you are working with on a regular basis is small fitting into the 4GB ssd buffer you will see performance the same or very close to a SSD. However as you fill up the SSD buffer and until the data gets destaged to the HDD, performance will shift back to what a traditional HDD provides. Likewise as data is read from the HDD into the SSD buffer, performance will increase. Oh, why not go all SSD? simple, cost!

My 500GB HHDDs give me for my needs performance comparable to SSDs while providing several times more space (which tends to be dormant) at a fraction of the cost. I also have SSDs that I use for some things where the focus is on as many IOPSs or as much performance as possible in a given amount of time. Hence if you have the need for speed and budget, go with the SSDs to reduce the number of HDDs needed. On the other hand, if you need to have a lot of data yet random performance needs, check out the HHDDs.

Cheers
gs

gregschulz
Explorer

Dwaddle is spot on with "...it depends..."

There is another aspect to bring into the discussion about I/O Operations Per second (IOPSs) which is the average size in kbytes of the work being done. As Dwaddle mentioned, various types of hard disk drives (HDDs), Hybrid HDDs (HHDDs) and solid state devices (SSDs) will have different IOP capabilties however those metrics are tied to work being done of a given size. The smaller the IO size, the higher the IOP and lower the bandwidth or throughput numbers, likewise, the larger the IO size, higher bandwidth or throughput will be seen with lower IOPs. Also keep latency or response time in mind which for IOPs should be lower for interactive or transactional activity.

Even though some HDDs can do 210 or more IOPSs, those IOPs are of a small size say 4Kbytes to 8Kbytes (check with specific vendors spec sheets) however depending on how those HDDs are attached to your computer or server will make a different. In addition to the type and speed of the interconnect (USB 2.0/3.0, SAS 3G/6G, 1GbE iSCSI, Fibre Channel, SATA, etc) the type of adapter, controller or storage system will also help or in some cases hinder the native drive performance. Generaly the controller, adapter or storage system should be neutral or boost performance with caching and other techniques. However it is possible where a controller or adapter or storage system can be implemeneted in a manner where the full drive potential is not realized. Likewise there are controllers, adapters and storage systems that are actually starved by not having enough HDDs to service IO which also tend to be a sign of the need for SSD or faster drives.

Also as Dwaddle mentions, RAID configuration can make a big differences with RAID 0 (stripe) being the fastest for reads and writes however it also provides no protection, in fact it actually introduces the risk of a single drive failure taking the entire stripeset off line. As a result it is typically only used for static or read data that can be rapidly restored from another disk, tape or other means. RAID 1 (mirroring) gives both good read and write performance however it has the highest capacity space overhead of the RAID levels. RAID 10 (1+0) and (0+1) stripe & mirror/mirror & stripe provide a good balance of performance with protection where data are stripped and mirrored (or mirrored and stripped) however at the expense of extra storage capacity. RAID5 is stripe with rotating parity and is very popular for concurrent reads however imposes a write penalty (hence need for battery backed write cache in controller/adapter/storage system) due to parity calculation operations. RAID 6 is popular for using large capacity low cost drives using dual parity with stripe to protect against a double drive failure. Another popular RAID option is RAID 4 which is what NetApp uses as a variation for their default proteciton in addition to the RAID DP (Douple Parity).

Reads will typcially be faster than writes, hence a higher percentage of smaller reads should yield a higher IOP rate (guess what vendors like to show for benchmark numbers? ) and hopefully lower latency. Likewise for high bandwidth or throughout to to show large number of MBytes or GBytes per second, the game is to use very large IOs of 64K, 128K or bigger such as would be seen with backup/restore, streaming video/audio, bulk data movement and other sequential operations. Cache in the controller/adapter/storage system can help particular on reads however there can also be write benefits.

There is much more to the topic, however hopefully that gives you more to think about. If you are interested or want to know more, check out my blog http://storageioblog.com or main website http://storageio.com where you can find articles, tips, videos, podcast, reports and other related content. In addition, in my books Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press) and Resilient Storage Networkss (Elseiver) I go into more discussion about IOPs, reads, writes, performance, workloads, benchmarking, storage systems and other related topics.

Cheers
gs

dwaddle
SplunkTrust
SplunkTrust

Sorry, but the best answer is "it depends on your workload." Read IOPS depends greatly on search activity. Write IOPS depends on indexing throughput. With the exception of caching, the disk subsystem doesn't care whether the 800 IOPS is read or write. A good cache can obviously greatly improve both read and write IOPS. (This includes operating system cache and controller cache).

Borrowing from http://www.symantec.com/connect/articles/getting-hang-iops (a most excellent article BTW), single drive IOPS is strongly related to avg drive seek time. Modern 15K RPM SAS drives can range from 175 to 210 IOPS per drive. Striping multiples IOPS, mirroing divides them.

If you can get ahold of 210 IOPS drives (which would have the lowest average seek time), then 4 of those in a RAID0 stripe gives you 840 IOPS (without cache). Put 8 of them into a RAID10 (4 striped mirrored against 4 more striped) is a solid 800IOPS solution. The lower-end 175 IOPS 15K RPM SAS drives in the same configuration could give 700 IOPS.

One substantial issue here is that most name-brand server vendors don't give you enough information behind a specific drive model. They may have two or three different drives they supply under the same generic "part number", each with different specs. (We have a big SAN array that uses Seagate and Maxtor drives side-by-side)

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...