I currently run 14 indexers over ~250 spinning disks with raidz2.
I've been running splunk over zfs for over 3 years.
No problems, in fact I initially ran indexers with several different backends and zfs was the most stable.
My main indexer is a 1U supermicro box with a cost of ~5k.
Box has 12 spinning media drives (raidz2), used Hitachi Ultrastar drives. 2 SSD's for bootup disks. Then install a PCI NVMe drive for your cache drive. (2-4 or so gigabytes per second speed). I'd recommend altering the default settings of zfs so it'll be more aggressive with using the NVMe drive.
Get at least 12 cores in order to remove the annoyance about Enterprise Security complaining <12 cores.
... View more