Re: Unacceptably Poor RAID1 Performance with Many CPU Cores

Hi,

在 2023/06/16 15:31, Ali Gholami Rudi 写道:

Ali Gholami Rudi [off-list ref] wrote:

quoted

Xiao Ni [off-list ref] wrote:

quoted

On Thu, Jun 15, 2023 at 10:06 PM Yu Kuai [off-list ref] wrote:

quoted

This looks familiar... Perhaps can you try to test with raid10 with
latest mainline kernel? I used to optimize spin_lock for raid10, and I
don't do this for raid1 yet... I can try to do the same thing for raid1
if it's valuable.

I do get improvements with raid10:

Without RAID (writing to /dev/ram0)
READ:  IOPS=15.8M BW=60.3GiB/s
WRITE: IOPS= 6.8M BW=27.7GiB/s

RAID1 (writing to /dev/md/test)
READ:  IOPS=518K BW=2028MiB/s
WRITE: IOPS=222K BW= 912MiB/s

RAID10 (writing to /dev/md/test)
READ:  IOPS=2033k BW=8329MB/s
WRITE: IOPS= 871k BW=3569MB/s

raid10 is about four times faster than raid1.

And this is perf's output for raid10:

+   97.33%     0.04%  fio      [kernel.kallsyms]       [k] entry_SYSCALL_64_after_hwframe
+   96.96%     0.02%  fio      [kernel.kallsyms]       [k] do_syscall_64
+   94.43%     0.03%  fio      [kernel.kallsyms]       [k] __x64_sys_io_submit
-   93.71%     0.04%  fio      [kernel.kallsyms]       [k] io_submit_one
    - 93.67% io_submit_one
       - 76.03% aio_write
          - 75.53% blkdev_write_iter
             - 68.95% blk_finish_plug
                - flush_plug_callbacks
                   - 68.93% raid10_unplug
                      - 64.31% __wake_up_common_lock
                         - 64.17% _raw_spin_lock_irqsave
                              native_queued_spin_lock_slowpath

This is unexpected, can you check if your kernel contain following
patch?

commit 460af1f9d9e62acce4a21f9bd00b5bcd5963bcd4
Author: Yu Kuai [off-list ref]
Date:   Mon May 29 21:11:06 2023 +0800

     md/raid1-10: limit the number of plugged bio

If so, can you retest with following patch?

diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index d0de8c9fb3cf..6fdd99c3e59a 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c

@@ -911,7 +911,7 @@ static void flush_pending_writes(struct r10conf *conf)

                 blk_start_plug(&plug);
                 raid1_prepare_flush_writes(conf->mddev->bitmap);
-               wake_up(&conf->wait_barrier);
+               wake_up_barrier(&conf->wait_barrier);

                 while (bio) { /* submit pending writes */
                         struct bio *next = bio->bi_next;

Thanks,
Kuai

                      - 4.43% submit_bio_noacct_nocheck
                         - 4.42% __submit_bio
                            - 2.28% raid10_end_write_request
                               - 0.82% raid_end_bio_io
                                    0.82% allow_barrier
                              2.09% brd_submit_bio
             - 6.41% __generic_file_write_iter
                - 6.08% generic_file_direct_write
                   - 5.64% __blkdev_direct_IO_async
                      - 4.72% submit_bio_noacct_nocheck
                         - 4.69% __submit_bio
                            - 4.67% md_handle_request
                               - 4.66% raid10_make_request
                                    2.59% raid10_write_one_disk
       - 16.14% aio_read
          - 15.07% blkdev_read_iter
             - 14.16% __blkdev_direct_IO_async
                - 11.36% submit_bio_noacct_nocheck
                   - 11.17% __submit_bio
                      - 5.89% md_handle_request
                         - 5.84% raid10_make_request
                            + 4.18% raid10_read_request
                      - 3.74% raid10_end_read_request
                         - 2.04% raid_end_bio_io
                              1.46% allow_barrier
                           0.55% mempool_free
                        1.39% brd_submit_bio
                - 1.33% bio_iov_iter_get_pages
                   - 1.00% iov_iter_get_pages
                      - __iov_iter_get_pages_alloc
                         - 0.85% get_user_pages_fast
                              0.75% internal_get_user_pages_fast
                  0.93% bio_alloc_bioset
               0.65% filemap_write_and_wait_range
+   88.31%     0.86%  fio      fio                     [.] thread_main
+   87.69%     0.00%  fio      [unknown]               [k] 0xffffffffffffffff
+   87.60%     0.00%  fio      fio                     [.] run_threads
+   87.31%     0.00%  fio      fio                     [.] do_io (inlined)
+   86.60%     0.32%  fio      libc-2.31.so            [.] syscall
+   85.87%     0.52%  fio      fio                     [.] td_io_queue
+   85.49%     0.18%  fio      fio                     [.] fio_libaio_commit
+   85.45%     0.14%  fio      fio                     [.] td_io_commit
+   85.37%     0.11%  fio      libaio.so.1.0.1         [.] io_submit
+   85.35%     0.00%  fio      fio                     [.] io_u_submit (inlined)
+   76.04%     0.01%  fio      [kernel.kallsyms]       [k] aio_write
+   75.54%     0.01%  fio      [kernel.kallsyms]       [k] blkdev_write_iter
+   68.96%     0.00%  fio      [kernel.kallsyms]       [k] blk_finish_plug
+   68.95%     0.00%  fio      [kernel.kallsyms]       [k] flush_plug_callbacks
+   68.94%     0.13%  fio      [kernel.kallsyms]       [k] raid10_unplug
+   64.41%     0.03%  fio      [kernel.kallsyms]       [k] _raw_spin_lock_irqsave
+   64.32%     0.01%  fio      [kernel.kallsyms]       [k] __wake_up_common_lock
+   64.05%    63.85%  fio      [kernel.kallsyms]       [k] native_queued_spin_lock_slowpath
+   21.05%     1.51%  fio      [kernel.kallsyms]       [k] submit_bio_noacct_nocheck
+   20.97%     1.18%  fio      [kernel.kallsyms]       [k] __blkdev_direct_IO_async
+   20.29%     0.03%  fio      [kernel.kallsyms]       [k] __submit_bio
+   16.15%     0.02%  fio      [kernel.kallsyms]       [k] aio_read
..

Thanks,
Ali

.