On 2020/5/29 6:07, Guoqing Jiang wrote:

On 5/27/20 3:19 PM, Yufen Yu wrote:

quoted

Hi, all

  For now, STRIPE_SIZE is equal to the value of PAGE_SIZE. That means, RAID5 will
  issus echo bio to disk at least 64KB when PAGE_SIZE is 64KB in arm64. However,
  filesystem usually issue bio in the unit of 4KB. Then, RAID5 will waste resource
  of disk bandwidth.

Could you explain a little bit about "waste resource"? Does it mean the chance for
full stripe write is limited because of  the incompatible between fs (4KB bio) and
raid5 (64KB stripe unit)?

Applications may request 4KB data, but RAID5 will issue 64KB size bio to disk, which
will waste disk bandwidth and more cpu time to compute xor. Detail performance data
can see in previous email:
https://www.spinics.net/lists/raid/msg64261.html

quoted

  To solve the problem, this patchset provide a new config CONFIG_MD_RAID456_STRIPE_SHIFT
  to let user config STRIPE_SIZE. The default value is 1, means 4096(1<<9).

  Normally, using default STRIPE_SIZE can get better performance. And NeilBrown have
  suggested just to fix the STRIPE_SIZE as 4096.But, out test result show that
  big value of STRIPE_SIZE may have better performance when size of issued IOs are
  mostly bigger than 4096. Thus, in this patchset, we still want to set STRIPE_SIZE
  as a configureable value.

I think it is better to define stripe size as 4K if it fits to generally scenario, and also
aligns with fs.

quoted

  In current implementation, grow_buffers() uses alloc_page() to allocate the buffers
  for each stripe_head. With the change, it means we allocate 64K buffers but just
  use 4K of them. To save memory, we try to 'compress' multiple buffers of stripe_head
  to only one real page. Detail shows in patch #2.

  To evaluate the new feature, we create raid5 device '/dev/md5' with 4 SSD disk
  and test it on arm64 machine with 64KB PAGE_SIZE.
  1) We format /dev/md5 with mkfs.ext4 and mount ext4 with default configure on
     /mnt directory. Then, trying to test it by dbench with command:
     dbench -D /mnt -t 1000 10. Result show as:
     'STRIPE_SHIFT = 64KB'
       Operation      Count    AvgLat    MaxLat
       ----------------------------------------
       NTCreateX    9805011     0.021    64.728
       Close        7202525     0.001     0.120
       Rename        415213     0.051    44.681
       Unlink       1980066     0.079    93.147
       Deltree          240     1.793     6.516
       Mkdir            120     0.004     0.007
       Qpathinfo    8887512     0.007    37.114
       Qfileinfo    1557262     0.001     0.030
       Qfsinfo      1629582     0.012     0.152
       Sfileinfo     798756     0.040    57.641
       Find         3436004     0.019    57.782
       WriteX       4887239     0.021    57.638
       ReadX        15370483     0.005    37.818
       LockX          31934     0.003     0.022
       UnlockX        31933     0.001     0.021
       Flush         687205    13.302   530.088
      Throughput 307.799 MB/sec  10 clients  10 procs  max_latency=530.091 ms
      -------------------------------------------------------
     'STRIPE_SIZE = 4KB'
       Operation      Count    AvgLat    MaxLat
       ----------------------------------------
       NTCreateX    11999166     0.021    36.380
       Close        8814128     0.001     0.122
       Rename        508113     0.051    29.169
       Unlink       2423242     0.070    38.141
       Deltree          300     1.885     7.155
       Mkdir            150     0.004     0.006
       Qpathinfo    10875921     0.007    35.485
       Qfileinfo    1905837     0.001     0.032
       Qfsinfo      1994304     0.012     0.125
       Sfileinfo     977450     0.029    26.489
       Find         4204952     0.019     9.361
       WriteX       5981890     0.019    27.804
       ReadX        18809742     0.004    33.491
       LockX          39074     0.003     0.025
       UnlockX        39074     0.001     0.014
       Flush         841022    10.712   458.848
      Throughput 376.777 MB/sec  10 clients  10 procs  max_latency=458.852 ms
      -------------------------------------------------------

What is the default io unit size of dbench?

Since dbench runs on ext4 filesystem, so I think most io size is about 4KB.

quoted

   2) We try to evaluate IO throughput for /dev/md5 by fio with config:
      [4KB randwrite]
      direct=1
      numjob=2
      iodepth=64
      ioengine=libaio
      filename=/dev/md5
      bs=4KB
      rw=randwrite
      [64KB write]
      direct=1
      numjob=2
      iodepth=64
      ioengine=libaio
      filename=/dev/md5
      bs=1MB
      rw=write
     The fio test result as follow:
                    +                   +
                    | STRIPE_SIZE(64KB) | STRIPE_SIZE(4KB)
      +----------------------------------------------------+
      4KB randwrite |     15MB/s        |      100MB/s
      +----------------------------------------------------+
      1MB write     |   1000MB/s        |      700MB/s
     The result show that when size of io is bigger than 4KB (64KB),
     64KB STRIPE_SIZE has much higher IOPS. But for 4KB randwrite, that
     means, size of io issued to device are smaller, 4KB STRIPE_SIZE
     have better performance.

The 4k rand write performance drops from 100MB/S to 15MB/S?! How about other
io sizes? Say 16k, 64K and 256K etc, it would be more convincing if 64KB stripe
has better performance than 4KB stripe overall.

Maybe I have not explain clearly. Here, the fio test result shows that 4KB
STRIPE_SIZE is not always have better performance. If applications request
IO size mostly are bigger than 4KB, likely 1MB in test, set STRIPE_SIZE with
a bigger value can get better performance.

So, we try to provide a configurable STRIPE_SIZE, rather than fix STRIPE_SIZE as 4096.

Thanks,
Yufen