For storage performance from a physical hardware perspective I would provide a few recommendations, most of them pretty straight forward.
Consider Flash Storage - somewhat obvious, but with costs decreasing this is an easy way to improved storage performance.
Blizzard did a really good, in depth analysis of the effects of flash and their testing in their direct attached storage (DAS) based environment at .conf 2016. Blizzard also noted that Bonnie++ is not necessarily the best indicator of storage performance.
Consider Scale-out Software Defined Storage (SDS) - one way to potentially improve storage efficiency, simplify management, and improve performance is spread IO across a bunch of different spindles managed by an SDS. Dell EMC has done work with Splunk to validate ScaleIO and vSAN on their platforms and the whitepapers are posted on [Splunk Partner Site][1]
Be Aware of Physical SAN Architecture - Splunk is a Scale-out application and if you are using physical SAN, even with Flash technology, have a good understanding of your storage architecture and utilization. Is your SAN architecture Scale-out or Scale-up?
Scale-up SAN is NOT bad, far from it, there are a number of very large deployments of Splunk that I know of that have been deployed on Scale-up SAN and it works really well. The thing you need to be cognizant of, in addition to the media and back end IO, is the front end capabilities of your fabric and service processors also need to be considered.
... View more