For storage performance from a physical hardware perspective I would provide a few recommendations, most of them pretty straight forward.
Consider Flash Storage - somewhat obvious, but with costs decreasing this is an easy way to improved storage performance.
Blizzard did a really good, in depth analysis of the effects of flash and their testing in their direct attached storage (DAS) based environment at .conf 2016. Blizzard also noted that Bonnie++ is not necessarily the best indicator of storage performance.
Consider Scale-out Software Defined Storage (SDS) - one way to potentially improve storage efficiency, simplify management, and improve performance is spread IO across a bunch of different spindles managed by an SDS. Dell EMC has done work with Splunk to validate ScaleIO and vSAN on their platforms and the whitepapers are posted on [Splunk Partner Site]
Be Aware of Physical SAN Architecture - Splunk is a Scale-out application and if you are using physical SAN, even with Flash technology, have a good understanding of your storage architecture and utilization. Is your SAN architecture Scale-out or Scale-up?
Scale-up SAN is NOT bad, far from it, there are a number of very large deployments of Splunk that I know of that have been deployed on Scale-up SAN and it works really well. The thing you need to be cognizant of, in addition to the media and back end IO, is the front end capabilities of your fabric and service processors also need to be considered.
... View more
The solutions guide Jenny refers to, “Using Splunk Enterprise with VxRail Appliances and Isilon for Analysis of Machine Data”, was published 3/1/2017: https://www.emc.com/collateral/service-overviews/h15699-splunk-vxrail-sg.pdf
Appendix A on page 87 provides some indicative vSAN performance data on VxRail to illustrate the linear scale of vSAN and VxRail for Splunk using IOmeter. One thing to note, as stated on page 88, in the test IOmeter was run on a VxRail Appliance hybrid configuration cluster to show the linear scalability of the VxRail cluster. This is a node that uses EFD cache and HDD media. While it is performant, it is not reflective of the higher performing all-flash VxRail configuration in the described scenarios. Dell EMC and Splunk’s recommendation is to use all-flash vSAN nodes. This is not necessarily an artifact of vSAN, though all-flash vSAN does have additional data services capabilities over traditional HDD/hybrid. If you see the presentation of Blizzard from .conf 2016 you see the general value of flash for Hot/Warm even in a direct attached storage environment with no software defined storage.
Hope this helps.
... View more