SAN vs Local Disk (LINUX BOX)

[robertlynch2020]

HI

I currently have my SPLUNK installed on a 500GB local LINUX Disk - RAID 10.
However we only have 10% left on this mount, i have gotten a SAN attached with RAID 10 read RAID6 write with 2000GB .

Is there any drawbacks from using local Disk vs SAN storage?

Am i better to make the hot/warm buckets on the local disk and cold on SAN?

Is there any test i can run to check what will happen if i move it over, or what should i ask the system admin?

Should i get my sys addmin to run the below commands, read and write. On the local disk vs SAN?
This will compare the IOPS and make sure that the SAN is as good as the local disk?

iostat -dx /dell425srv2 | grep /dell425srv2 | awk '{ print $4; }'
iostat -dx /dell425srv2 | grep /dell425srv2 | awk '{ print $5; }'

Thanks
Robert

[woodcock]

Q1: Is there any drawbacks from using local Disk vs SAN storage?
A1: Yes. You MUST use RAID (0 or 10) and you MUST BE PROACTIVE to monitor for disk failures. YOU have to do this; you WILL have disk failures and if you have a second before you fix the first, you will lose all data.

Q2: Am i better to make the hot/warm buckets on the local disk and cold on SAN?
A2: Yes, that is the normal way: Hot/Warm on fast local (direct-attached) disk, and cold on SAN.

Q3: Is there any test i can run to check what will happen if i move it over, or what should i ask the system admin?
A3: Check the reference spec for minimum disk iops and use bonnie++ or other tool to ensure that you got them.

Q4: Should i get my sys addmin to run the below commands, read and write. On the local disk vs SAN?
A4: You really should use a tool built for this purpose, although that is a start.

Q5: This will compare the IOPS and make sure that the SAN is as good as the local disk?
A5: The SAN need not be (probably cannot be) as good as the Warm local.

[Richfez]

A couple of important points:

First, the "speed" of the SAN may or may not be fine right now, but will it be configured in such a way as to not starve Splunk for IOPS during, say, nightly backups? It's a shared resource and Splunk often needs the MOST resources right at the time where the SAN is most loaded up with IO and is the least capable of providing it.

Second, what does "RAID 10 read RAID6 write" mean? RAID 10 is not RAID 6. They handle data differently, have different performance and space characteristics and most importantly are actual physical arrangements of disks. It's possible you have RAID 60, or the SAN uses some useful shenanigans to increase write performance, but in the end they are physically different things. And, R6 would be the problem if there is one. R10 on the SAN may be totally fine depending on other loads. R6 COULD be OK on some fancy SANs that do a ton of caching and whatnot.

But lastly, and good news -

Check IOPS however you want. Check the interwebs for ways to use bonnie++ or one of the other utilities, there are even answers around that address this already, but I suspect the SAN will be a huge performance boost. You may want to do something different long term, but it might be a useful boost right now.

BEST: can you reconfigure the SAN to just be RAID 10 the normal way, no R6 involved?

[robertlynch2020]

Hi

Thanks for the advice, in the end we copied an SPLUNK over to the server and we tested it.

The speed was the same so we moved the original copy over.

Thanks
Robert Lynch

[skalliger]

There is a tool called bonnie++ (https://splunkbase.splunk.com/app/3002/) to measure IOPS.
It's a good practice to place the path of your hot/warm buckets on the fastest disk, while you can remain the cold/frozen ones on the slower disk/storage.

So, depends whether your SAN is faster than your local storage or not.

Skalli

[robertlynch2020]

Hi - Thanks for this.

However i was hoping for a quick command to be able to tell me what is the best option here.
At the moment i only have one INDEX in the one install i have, to use BONNIE++ i have to create more indexers etc..

Cheers
Robert