I have a situation with Splunk and how things are configured. We have multiple indexers and they have local SSD drives configured or the hot/warm mount point and we use our FLASH SAN attached over FibreChannel for the cold mount points. We need to expand the hot mount point and this will require purchasing a fair amount of equipment to expand the SSD that is installed. We could easily put it on the SAN and move the SSD to the SAN. What are the recommendations? Can we use SAN instead of SSD? Here are the bonnie++ and fio outputs for both the SAN and SSD. nohup /usr/local/bin/bonnie++ -d /ssd/splunk -x 1 -u root:root -q -f > /ssd_disk_io_test.csv 2> /ssd_disk_io_test.err < /dev/null FLASH SAN: Sequential Output Sequential Input Random Seeks Sequential Create Random Create Size:Chunk Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU SERVER 505G 528475 99 252234 75 539415 83 11333.2 69 16 12646 99 +++++ +++ +++++ +++ 14164 99 +++++ +++ +++++ +++ SSD: Sequential Output Sequential Input Random Seeks Sequential Create Random Create Size:Chunk Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU SERVER 505G 505646 99 218685 62 504329 74 +++++ +++ 16 12085 99 +++++ +++ +++++ +++ 12162 99 +++++ +++ +++++ +++ nohup /usr/local/bin/bonnie++ -d /ssd/splunk -s 516696 -u root:root -fb > /ssd_bonnie-seth.csv 2> /ssd_bonnie-seth.err < /dev/null NON-SSD: Sequential Output Sequential Input Random Seeks Sequential Create Random Create Size:Chunk Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU SERVER 516696M 509275 97 241009 74 538182 81 7549.2 57 16 616 17 +++++ +++ 780 6 556 12 +++++ +++ 884 7 SSD: Sequential Output Sequential Input Random Seeks Sequential Create Random Create Size:Chunk Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU / sec % CPU SERVER 516696M 501271 98 218604 62 523329 75 8428.1 60 16 501 13 +++++ +++ 570 4 499 15 +++++ +++ 579 4 Here is the output of FIO for SSD: 4k_benchmark: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=128 ... fio-3.5 Starting 12 processes 4k_benchmark: Laying out IO file (1 file / 102400MiB) 4k_benchmark: (groupid=0, jobs=12): err= 0: pid=29545: Thu May 3 00:53:33 2018 read: IOPS=1307, BW=1308MiB/s (1371MB/s)(38.4GiB/30063msec) slat (usec): min=178, max=445349, avg=9130.87, stdev=15882.69 clat (msec): min=17, max=2401, avg=1139.06, stdev=330.41 lat (msec): min=25, max=2420, avg=1148.20, stdev=331.67 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 592], 10.00th=[ 718], 20.00th=[ 869], | 30.00th=[ 978], 40.00th=[ 1070], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1318], 80.00th=[ 1418], 90.00th=[ 1552], 95.00th=[ 1670], | 99.00th=[ 1905], 99.50th=[ 2022], 99.90th=[ 2232], 99.95th=[ 2299], | 99.99th=[ 2366] bw ( KiB/s): min= 4096, max=329728, per=8.27%, avg=110678.16, stdev=45552.32, samples=700 iops : min= 4, max= 322, avg=107.93, stdev=44.42, samples=700 lat (msec) : 20=0.01%, 50=0.03%, 100=0.20%, 250=0.62%, 500=2.10% lat (msec) : 750=8.87%, 1000=20.50% cpu : usr=0.20%, sys=7.40%, ctx=53123, majf=0, minf=1233 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.5%, 32=1.0%, >=64=98.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=39314,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=1308MiB/s (1371MB/s), 1308MiB/s-1308MiB/s (1371MB/s-1371MB/s), io=38.4GiB (41.2GB), run=30063-30063msec Disk stats (read/write): dm-10: ios=78343/9, merge=0/0, ticks=4005205/515, in_queue=4059709, util=99.69%, aggrios=78628/7, aggrmerge=0/3, aggrticks=4012840/24755, aggrin_queue=4037190, aggrutil=99.66% sdb: ios=78628/7, merge=0/3, ticks=4012840/24755, in_queue=4037190, util=99.66% Here is the output for fio for FLASH SAN: 4k_benchmark: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=128 ... fio-3.5 Starting 12 processes 4k_benchmark: Laying out IO file (1 file / 102400MiB) 4k_benchmark: (groupid=0, jobs=12): err= 0: pid=54555: Wed Apr 18 10:54:55 2018 read: IOPS=2462, BW=2463MiB/s (2582MB/s)(72.2GiB/30007msec) slat (usec): min=243, max=178818, avg=4850.89, stdev=9427.35 clat (usec): min=623, max=1639.7k, avg=604472.15, stdev=172064.11 lat (usec): min=1986, max=1751.5k, avg=609326.97, stdev=173108.88 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 393], 10.00th=[ 435], 20.00th=[ 481], | 30.00th=[ 514], 40.00th=[ 550], 50.00th=[ 575], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 709], 90.00th=[ 818], 95.00th=[ 969], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1318], 99.95th=[ 1334], | 99.99th=[ 1401] bw ( KiB/s): min= 4104, max=416193, per=8.42%, avg=212218.39, stdev=58840.35, samples=701 iops : min= 4, max= 406, avg=206.70, stdev=57.47, samples=701 lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.04%, 50=0.11% lat (msec) : 100=0.19%, 250=0.70%, 500=24.14%, 750=60.18%, 1000=10.62% cpu : usr=0.27%, sys=15.52%, ctx=101062, majf=0, minf=1234 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=0.5%, >=64=99.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=73895,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=72.2GiB (77.5GB), run=30007-30007msec Disk stats (read/write): dm-11: ios=146961/5, merge=0/0, ticks=276683/6, in_queue=277077, util=99.19%, aggrios=73895/3, aggrmerge=0/0, aggrticks=138910/3, aggrin_queue=139108, aggrutil=99.17% dm-2: ios=0/4, merge=0/0, ticks=0/2, in_queue=2, util=0.01%, aggrios=0/3, aggrmerge=0/1, aggrticks=0/2, aggrin_queue=2, aggrutil=0.01% dm-0: ios=0/3, merge=0/1, ticks=0/2, in_queue=2, util=0.01%, aggrios=0/2, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% sdc: ios=0/2, merge=0/0, ticks=0/0, in_queue=0, util=0.00% sdf: ios=0/2, merge=0/0, ticks=0/1, in_queue=1, util=0.00% dm-3: ios=147790/3, merge=0/0, ticks=277820/4, in_queue=278214, util=99.17%, aggrios=147790/3, aggrmerge=0/0, aggrticks=279775/4, aggrin_queue=279008, aggrutil=99.11% dm-1: ios=147790/3, merge=0/0, ticks=279775/4, in_queue=279008, util=99.11%, aggrios=73895/1, aggrmerge=0/0, aggrticks=129019/0, aggrin_queue=128825, aggrutil=98.68% sdd: ios=73894/2, merge=0/0, ticks=125568/0, in_queue=125371, util=98.66% sdg: ios=73896/1, merge=0/0, ticks=132470/0, in_queue=132279, util=98.68%
... View more