「fio」- 块存储,性能测试

  CREATED BY JENKINSBOT

环境准备工作

磁盘填充测试数据

Q:为什么要进行数据填充?
A:针对空块含数据的块,当从两者中读取数据读时,将会得到不同的性能结果。所以为使结果更加准确,我们将对块设备进行数据填充;

fio --name=fill_disk                                       \
  --filename=/dev/sdc --filesize=29313144Ki                \
  --ioengine=libaio --direct=1 --verify=0 --randrepeat=0   \
  --bs=128K --iodepth=64 --rw=randwrite

# 该命令参数将完成数据填充的任务。鉴于其参数并不符合性能测试要求,所以我们不无需关注其输出的报告;

存储测试配置信息

命令选项如下,其为各个测试场景中的通用选项:

–filename=/dev/sdc:物理磁盘路径:/dev/sdc
–filesize=29313144Ki:我们的磁盘大小,30016659456 / 1024 = 29313144Ki (~28G)

–time_based:基于时间来进行基准测试,即任务仅云新特定时间;
–runtime=1m:测试任务仅运行一分钟,得到测试结果即可,无需长时间运行;
–ramp_time=2s:当使得性能趋于稳定后,再输出性能测试结果(在 2s 后);

–ioengine=libaio:使用异步 IO 引擎,以模拟更多负载;
–direct=1:使用 non-buffered I/O,将数据直接写入物理磁盘。
–verify=0:无需要对写入的数据进行验证
–randrepeat=0:随机产生数据;

Bandwidth(带宽)

–bs=1M:块大小对带宽的测试结果是有影响的,具体块大小取决于业务场景;
–iodepth=64:需要足够的 I/O depth,以产生足够多个 IO 负载,我们这里 I/O depth 为 64;
–rw=write:需要通过 Seq. r/w 来实现带宽测试,而 Rnd. r/w 将消耗更多时间进行寻道,进而无法得到准确的结果;
–numjobs=16:需要创建多个进程,以产生足够多的 IO 负载,我们这里创建 16 个进程;
–offset_increment=1832071Ki:将文件分为 16 部分(29313144÷16),使得每个进程仅写入自己所属的部分;
–group_reporting,能够将多个进程的输出结果进行聚合显示,而非显示每个进程的测试报告;

写入带宽(Seq. Bandwidth)

fio --name=write_bandwidth_test                            \
  --filename=/dev/sdc --filesize=29313144Ki                \
  --time_based --ramp_time=2s --runtime=1m                 \
  --ioengine=libaio --direct=1 --verify=0 --randrepeat=0   \
  --bs=1M --iodepth=64 --rw=write --numjobs=16 --offset_increment=1832071Ki --group_reporting

write_bandwidth_test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 16 processes
Jobs: 15 (f=15): [W(1),f(1),W(1),f(3),_(1),W(9)][52.4%][w=329MiB/s][w=329 IOPS][eta 00m:59s]
write_bandwidth_test: (groupid=0, jobs=16): err= 0: pid=709992: Mon Jan  9 19:00:32 2023
  write: IOPS=17, BW=20.3MiB/s (21.3MB/s)(1267MiB/62387msec); 0 zone resets
    slat (usec): min=30, max=11984k, avg=894433.93, stdev=1674325.44
    clat (msec): min=528, max=63688, avg=41448.14, stdev=15826.27
     lat (msec): min=1536, max=64441, avg=42249.77, stdev=15302.83
    clat percentiles (msec):
     |  1.00th=[  944],  5.00th=[ 8154], 10.00th=[16845], 20.00th=[17113],
     | 30.00th=[17113], 40.00th=[17113], 50.00th=[17113], 60.00th=[17113],
     | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113],
     | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min=30720, max=122912, per=100.00%, avg=62313.85, stdev=2378.99, samples=123
   iops        : min=   30, max=  120, avg=60.84, stdev= 2.32, samples=123
  lat (msec)   : 750=0.55%, 1000=0.73%, 2000=0.55%, >=2000=114.30%
  cpu          : usr=0.00%, sys=0.00%, ctx=583, majf=0, minf=930
  IO depths    : 1=0.0%, 2=0.1%, 4=1.0%, 8=7.7%, 16=20.5%, 32=46.9%, >=64=23.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.2%, 8=0.0%, 16=0.0%, 32=0.0%, 64=5.8%, >=64=0.0%
     issued rwts: total=0,1091,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=20.3MiB/s (21.3MB/s), 20.3MiB/s-20.3MiB/s (21.3MB/s-21.3MB/s), io=1267MiB (1329MB), run=62387-62387msec

Disk stats (read/write):
  sdc: ios=99/1349, merge=0/1, ticks=64/3873284, in_queue=3873347, util=99.87%

通过创建线程来执行测试:

# --numjobs=4 --thread
fio --name=read_bandwidth_test                             \
  --filename=/dev/sdc --filesize=29313144Ki                \
  --time_based --ramp_time=2s --runtime=1m                 \
  --ioengine=libaio --direct=1 --verify=0 --randrepeat=0   \
  --bs=1M --iodepth=64 --rw=write --numjobs=4 --thread --offset_increment=1832071Ki

...

读取带宽(Rnd. Bandwidth)

fio --name=read_bandwidth_test                             \
  --filename=/dev/sdc --filesize=29313144Ki                \
  --time_based --ramp_time=2s --runtime=1m                 \
  --ioengine=libaio --direct=1 --verify=0 --randrepeat=0   \
  --bs=1M --iodepth=64 --rw=read --numjobs=16 --offset_increment=1832071Ki --group_reporting

read_bandwidth_test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 16 processes
Jobs: 2 (f=0): [_(8),f(1),_(6),f(1)][100.0%][r=1011MiB/s][r=1011 IOPS][eta 00m:00s]           
read_bandwidth_test: (groupid=0, jobs=16): err= 0: pid=1306048: Tue Jan 10 15:05:02 2023
  read: IOPS=115, BW=132MiB/s (138MB/s)(8017MiB/60859msec)
    slat (usec): min=15, max=1369.9k, avg=137654.31, stdev=178021.94
    clat (msec): min=274, max=15604, avg=8256.65, stdev=2178.45
     lat (msec): min=476, max=15747, avg=8395.81, stdev=2187.67
    clat percentiles (msec):
     |  1.00th=[  844],  5.00th=[ 3675], 10.00th=[ 6141], 20.00th=[ 7148],
     | 30.00th=[ 7617], 40.00th=[ 8020], 50.00th=[ 8423], 60.00th=[ 8792],
     | 70.00th=[ 9194], 80.00th=[ 9731], 90.00th=[10537], 95.00th=[11476],
     | 99.00th=[13087], 99.50th=[14026], 99.90th=[15368], 99.95th=[15503],
     | 99.99th=[15637]
   bw (  KiB/s): min=32335, max=385194, per=90.40%, avg=121946.96, stdev=4406.21, samples=1881
   iops        : min=   29, max=  376, avg=119.00, stdev= 4.30, samples=1881
  lat (msec)   : 500=0.21%, 750=0.71%, 1000=0.41%, 2000=1.57%, >=2000=111.13%
  cpu          : usr=0.00%, sys=0.02%, ctx=3682, majf=0, minf=8628
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.3%, >=64=99.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.2%, >=64=0.0%
     issued rwts: total=7023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=8017MiB (8406MB), run=60859-60859msec

Disk stats (read/write):
  sdc: ios=9092/0, merge=80/0, ticks=4272590/0, in_queue=4272590, util=99.89%

通过创建线程来执行测试:

# --numjobs=4 --thread
fio --name=read_bandwidth_test                             \
  --filename=/dev/sdc --filesize=29313144Ki                \
  --time_based --ramp_time=2s --runtime=1m                 \
  --ioengine=libaio --direct=1 --verify=0 --randrepeat=0   \
  --bs=1M --iodepth=64 --rw=read --numjobs=4 --thread --offset_increment=1832071Ki

...

IOPS(每秒 IO 数)

–rw=randwrite:写入 IOPS 需要使用 Rnd. Write 实现;读取 IOPS 需要使用 Rnd. Read 实现;

–iodepth=256:要实现最大 IOPS,必须维护一个深 I/O 队列。例如,如果写入延迟为 1ms,则 VM 最多可为每个运行中的 I/O 实现 1k IOPS。要达到 1.5w Write IOPS,VM 必须保持至少 15 个 I/O 处于运行状态。如果磁盘和虚拟机能够达到 3w Write IOPS,则运行中的 I/O 数量必须至少为 30 I/O

–bs=4K:采用 4K 块大小是基于一些默认规则:底层存储的块大小为 4K,其亦以 4K 作为读写单位,并且我们要测试的是存储单个块(页)的 IOPS 速度,此外厂商的给出的 IOPS 指标也是基于 4K 块。如果 I/O blocksize > 4 KB,那么在达到存储 IOPS 限制之前会先达到 bandwidth 限制,导致测试结果不准确。严谨地讲,具体 blocksize 取决于对业务数据的采集分析,我们这里仅是通用测试;

写入次数(Rnd. IOPS)

fio --name=write_iops_test                                 \
  --filename=/dev/sdc --filesize=29313144Ki                \
  --time_based --ramp_time=2s --runtime=1m                 \
  --ioengine=libaio --direct=1 --verify=0 --randrepeat=0   \
  --bs=4K --iodepth=256 --rw=randwrite

write_iops_test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=3212KiB/s][w=803 IOPS][eta 00m:00s]
write_iops_test: (groupid=0, jobs=1): err= 0: pid=1369609: Tue Jan 10 17:13:07 2023
  write: IOPS=716, BW=2872KiB/s (2941kB/s)(168MiB/60040msec); 0 zone resets
    slat (nsec): min=1719, max=2631.2M, avg=1454024.84, stdev=23988982.61
    clat (msec): min=37, max=3401, avg=360.12, stdev=346.97
     lat (msec): min=74, max=3401, avg=361.57, stdev=347.81
    clat percentiles (msec):
     |  1.00th=[  236],  5.00th=[  255], 10.00th=[  262], 20.00th=[  271],
     | 30.00th=[  288], 40.00th=[  296], 50.00th=[  305], 60.00th=[  309],
     | 70.00th=[  313], 80.00th=[  338], 90.00th=[  368], 95.00th=[  460],
     | 99.00th=[ 3004], 99.50th=[ 3239], 99.90th=[ 3306], 99.95th=[ 3306],
     | 99.99th=[ 3339]
   bw (  KiB/s): min=  520, max= 4120, per=100.00%, avg=3205.08, stdev=553.31, samples=107
   iops        : min=  130, max= 1030, avg=801.19, stdev=138.32, samples=107
  lat (msec)   : 50=0.01%, 100=0.07%, 250=3.09%, 500=92.34%, 750=1.10%
  lat (msec)   : 1000=0.66%, 2000=1.56%, >=2000=1.39%
  cpu          : usr=0.14%, sys=0.22%, ctx=3340, majf=0, minf=58
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,43022,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=2872KiB/s (2941kB/s), 2872KiB/s-2872KiB/s (2941kB/s-2941kB/s), io=168MiB (177MB), run=60040-60040msec

Disk stats (read/write):
  sdc: ios=51/43050, merge=0/2, ticks=97/3861716, in_queue=3861814, util=99.92%

读取次数(Rnd. IOPS)

fio --name=read_iops_test                                  \
  --filename=/dev/sdc --filesize=29313144Ki                \
  --time_based --ramp_time=2s --runtime=1m                 \
  --ioengine=libaio --direct=1 --verify=0 --randrepeat=0   \
  --bs=4K --iodepth=256 --rw=randread

read_iops_test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=14.5MiB/s][r=3712 IOPS][eta 00m:00s]
read_iops_test: (groupid=0, jobs=1): err= 0: pid=1383031: Tue Jan 10 17:40:23 2023
  read: IOPS=3722, BW=14.6MiB/s (15.3MB/s)(874MiB/60013msec)
    slat (nsec): min=1615, max=4834.7k, avg=267219.55, stdev=733156.54
    clat (msec): min=12, max=134, avg=68.47, stdev= 5.54
     lat (msec): min=15, max=134, avg=68.73, stdev= 5.58
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   63], 10.00th=[   63], 20.00th=[   64],
     | 30.00th=[   66], 40.00th=[   67], 50.00th=[   69], 60.00th=[   69],
     | 70.00th=[   71], 80.00th=[   72], 90.00th=[   74], 95.00th=[   77],
     | 99.00th=[   90], 99.50th=[   96], 99.90th=[  109], 99.95th=[  114],
     | 99.99th=[  121]
   bw (  KiB/s): min=14813, max=14992, per=99.94%, avg=14897.39, stdev=39.15, samples=120
   iops        : min= 3703, max= 3748, avg=3724.31, stdev= 9.79, samples=120
  lat (msec)   : 20=0.01%, 50=0.05%, 100=99.73%, 250=0.32%
  cpu          : usr=0.57%, sys=1.11%, ctx=27997, majf=0, minf=58
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=223392,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=14.6MiB/s (15.3MB/s), 14.6MiB/s-14.6MiB/s (15.3MB/s-15.3MB/s), io=874MiB (916MB), run=60013-60013msec

Disk stats (read/write):
  sdc: ios=230814/0, merge=4/0, ticks=3746205/0, in_queue=3746204, util=99.87%

Latency(延迟)

–rw=randwrite:通过 Rnd. Write 测试写入延迟;通过 Rnd. Read 测试读取延迟;

–iodepth=4:在测试 I/O 延迟时,不得达到最大 bandwidth 或 IOPS 限制,否则观察到的延迟不会反映实际的 I/O 延迟。例如,假设在 I/O depth = 30 时,将达到 IOPS 限制。如果将 I/O depth 增加一倍,此时由于总 IOPS 保持不变,将导致报告的 I/O 延迟加倍;

–bs=4K:依旧使用 4K 块大小,测试 4K 块的 IO 延迟;

写入延迟(Rnd. Latency)

fio --name=write_latency_test                              \
  --filename=/dev/sdc --filesize=29313144Ki                \
  --time_based --ramp_time=2s --runtime=1m                 \
  --ioengine=libaio --direct=1 --verify=0 --randrepeat=0   \
  --bs=4K --iodepth=4 --rw=randwrite

write_latency_test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=3712KiB/s][w=928 IOPS][eta 00m:00s] 
write_latency_test: (groupid=0, jobs=1): err= 0: pid=1390106: Tue Jan 10 17:54:35 2023
  write: IOPS=2089, BW=8358KiB/s (8558kB/s)(490MiB/60018msec); 0 zone resets
    slat (nsec): min=2032, max=56159, avg=3196.80, stdev=2166.19
    clat (usec): min=13, max=2635.4k, avg=1972.89, stdev=41852.04
     lat (usec): min=24, max=2635.4k, avg=1976.15, stdev=41852.03
    clat percentiles (usec):
     |  1.00th=[     25],  5.00th=[     34], 10.00th=[     35],
     | 20.00th=[     39], 30.00th=[     41], 40.00th=[     43],
     | 50.00th=[     51], 60.00th=[     52], 70.00th=[     54],
     | 80.00th=[     64], 90.00th=[   1844], 95.00th=[   1991],
     | 99.00th=[  38011], 99.50th=[  38536], 99.90th=[  95945],
     | 99.95th=[ 102237], 99.99th=[2600469]
   bw (  KiB/s): min=  128, max=57194, per=100.00%, avg=11151.30, stdev=17506.45, samples=90
   iops        : min=   32, max=14298, avg=2787.73, stdev=4376.61, samples=90
  lat (usec)   : 20=0.01%, 50=49.41%, 100=37.76%, 250=0.30%, 500=0.01%
  lat (usec)   : 750=0.15%, 1000=0.38%
  lat (msec)   : 2=7.52%, 4=1.64%, 10=0.01%, 20=0.01%, 50=2.67%
  lat (msec)   : 100=0.09%, 250=0.03%, 1000=0.01%, >=2000=0.02%
  cpu          : usr=0.40%, sys=0.75%, ctx=67420, majf=0, minf=58
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,125402,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=8358KiB/s (8558kB/s), 8358KiB/s-8358KiB/s (8558kB/s-8558kB/s), io=490MiB (514MB), run=60018-60018msec

Disk stats (read/write):
  sdc: ios=51/131422, merge=0/0, ticks=68/252354, in_queue=252422, util=99.91%

读取延迟(Rnd. Latency)

fio --name=read_latency_test                               \
  --filename=/dev/sdc --filesize=29313144Ki                \
  --time_based --ramp_time=2s --runtime=1m                 \
  --ioengine=libaio --direct=1 --verify=0 --randrepeat=0   \
  --bs=4K --iodepth=4 --rw=randread

read_latency_test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=14.5MiB/s][r=3711 IOPS][eta 00m:00s]
read_latency_test: (groupid=0, jobs=1): err= 0: pid=1392701: Tue Jan 10 17:59:50 2023
  read: IOPS=3712, BW=14.5MiB/s (15.2MB/s)(870MiB/60001msec)
    slat (nsec): min=2149, max=81744, avg=5073.49, stdev=2086.35
    clat (usec): min=370, max=1333, avg=1071.11, stdev=59.67
     lat (usec): min=374, max=1337, avg=1076.30, stdev=59.49
    clat percentiles (usec):
     |  1.00th=[  906],  5.00th=[  996], 10.00th=[ 1012], 20.00th=[ 1037],
     | 30.00th=[ 1057], 40.00th=[ 1057], 50.00th=[ 1074], 60.00th=[ 1090],
     | 70.00th=[ 1090], 80.00th=[ 1106], 90.00th=[ 1123], 95.00th=[ 1139],
     | 99.00th=[ 1172], 99.50th=[ 1188], 99.90th=[ 1221], 99.95th=[ 1237],
     | 99.99th=[ 1270]
   bw (  KiB/s): min=14800, max=14928, per=100.00%, avg=14856.06, stdev=24.21, samples=120
   iops        : min= 3700, max= 3732, avg=3713.99, stdev= 6.03, samples=120
  lat (usec)   : 500=0.12%, 750=0.37%, 1000=4.79%
  lat (msec)   : 2=94.72%
  cpu          : usr=1.13%, sys=2.42%, ctx=222455, majf=0, minf=58
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=222774,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=14.5MiB/s (15.2MB/s), 14.5MiB/s-14.5MiB/s (15.2MB/s-15.2MB/s), io=870MiB (912MB), run=60001-60001msec

Disk stats (read/write):
  sdc: ios=230159/0, merge=0/0, ticks=244086/0, in_queue=244085, util=99.88%

存储性能测试总结

Q:针对如上测试结果,该硬盘的性能指标是好还是坏?
A:我们无法单纯的去讨论结果的好坏,需要(1)将其与厂商给到的基准测试结果进行对比,了解差距;(2)要与业务需求进行对比,以了解该磁盘能否满足业务需求;

参考文献

Compute Engine Documentation/Benchmarking persistent disk performance
Benchmarking persistent disk performance
Alibaba Cloud Community/How to Use FIO to Test the IO Performance of ECS Local SSD and ESSD (Part 1)