What kind of performance gain by moving from SAN to local storage?

[usd0872]

We have 4 indexers in an index cluster, processing up to 400GB/d. Data volume is growing, thus we need more horse-power. For indexers, disk-I/O is one of the biggest performance parameters. Due to company rules, all our disks are SAN storage. After long discussions, management is now at least making calculations, what local storage for our indexers might cost.

Has anybody some experience with going from SAN storage on an index cluster to local storage?
- Have you gone to normal disks or SSD?
- What kind of performance gain did you see? (eg. xx% less indexers required for the same data volume)
- Can you confirm my feeling, that scaling with local disks is more linear than with SAN disks, because, as you add indexers, you always go to a new set of disks with local storage and with SAN storage you increase the load on the same backend infrastructure?

[gcusello]

Hi usd0872,
in my experience I encountered projects with NAS (terrible!), SAN (good) and local (better) I didn't experienced with SSD but I think that it is also better!

To understand how much it's better I suggest to use a tool like Bonnie++ (or another one!) to measure IOPS of your storage aligning with the Splunk storage requirements (see http://docs.splunk.com/Documentation/Splunk/6.5.2/Capacity/Referencehardware).

About SSD, see http://blogs.splunk.com/2012/05/10/quantifying-the-benefits-of-splunk-with-ssds/.

About performance gain, Splunk says to calculate a 30% of configuration (RAM, CPU, ...) using virtual disks, so you should wait for something near.

About your last question, I can confirm your feeling, that scaling with local disks is more linear than with SAN disks, because, as you add indexers, you always go to a new set of disks with local storage and with SAN storage you increase the load on the same backend infrastructure.

Bye.
Giuseppe

[Rob2520]

@gcusello Thanks for the post. Do we have to stop Splunk or put cluster master in maintenance mode or any other change during the migration from SAN to local disks?

[gcusello]

Hi @usd0872,
no, you have to migrate indexers one by one following documentation at https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Moveanindex .

Ciao.
Giuseppe