On 1/24/25 21:30, Ming Lei wrote:

quoted

1 queue:
========
                              +-------------------+-------------------+
                              | ublk (IOPS / BW)  | zloop (IOPS / BW) |
 +----------------------------+-------------------+-------------------+
 | QD=1,    4K rnd wr, 1 job  | 11.7k / 47.8 MB/s | 15.8k / 53.0 MB/s |
 | QD=32,   4K rnd wr, 8 jobs | 63.4k / 260 MB/s  | 101k / 413 MB/s   |

I can't reproduce the above two, actually not observe obvious difference
between rublk/zoned and zloop in my test VM.

I am using bare-metal machines for these tests as I do not want any
noise from a VM/hypervisor in the numbers. And I did say that this is with a
tweaked version of zloop that I have not posted yet (I was waiting for rc1 to
repost as a rebase is needed to correct a compilation failure du to the nomerge
tage set flag being removed). I am attaching the patch I used here (it applies
on top of current Linus tree)

Maybe rublk works at debug mode, which reduces perf by half usually.
And you need to add device via 'cargo run -r -- add zoned' for using
release mode.

Well, that is not an obvious thing for someone who does not know rust well. The
README file of rublk also does not mention that. So no, I did not run it like
this. I followed the README and call rublk directly. It would be great to
document that.

Actually there is just single io_uring_enter() running in each ublk queue
pthread, perf should be similar with kernel IO handling, and the main extra
load is from the single syscall kernel/user context switch and IO data copy,
and data copy effect can be neglected in small io size usually(< 64KB).

quoted

 | QD=32, 128K rnd wr, 1 job  | 5008 / 656 MB/s   | 5993 / 786 MB/s   |
 | QD=32, 128K seq wr, 1 job  | 2636 / 346 MB/s   | 5393 / 707 MB/s   |

ublk 128K BS may be a little slower since there is one extra copy.

Here are newer numbers running rublk as you suggested (using cargo run -r).
The backend storage is on an XFS file system using a PCI gen4 4TB M.2 SSD that
is empty (the FS is empty on start). The emulated zoned disk has a capacity of
512GB with sequential zones only of 256 MB (that is, there are 2048
zones/files). Each data point is from a 1min run of fio.

On a 8-cores Intel Xeon test box, which has PCI gen 3 only, I get:

Single queue:
=============
                              +-------------------+-------------------+
                              | ublk (IOPS / BW)  | zloop (IOPS / BW) |
 +----------------------------+-------------------+-------------------+
 | QD=1,    4K rnd wr, 1 job  | 2859 / 11.7 MB/s  | 5535 / 22.7 MB/s  |
 | QD=32,   4K rnd wr, 8 jobs | 24.5k / 100 MB/s  | 24.6k / 101 MB/s  |
 | QD=32, 128K rnd wr, 1 job  | 14.9k / 1954 MB/s | 19.6k / 2571 MB/s |
 | QD=32, 128K seq wr, 1 job  | 1516 / 199 MB/s   | 10.6k / 1385 MB/s |
 +----------------------------+-------------------+-------------------+

8 queues:
=========
                              +-------------------+-------------------+
                              | ublk (IOPS / BW)  | zloop (IOPS / BW) |
 +----------------------------+-------------------+-------------------+
 | QD=1,    4K rnd wr, 1 job  | 5387 / 22.1 MB/s  | 5436 / 22.3 MB/s  |
 | QD=32,   4K rnd wr, 8 jobs | 16.4k / 67.0 MB/s | 26.3k / 108 MB/s  |
 | QD=32, 128K rnd wr, 1 job  | 6101 / 800 MB/s   | 19.8k / 2591 MB/s |
 | QD=32, 128K seq wr, 1 job  | 3987 / 523 MB/s   | 10.6k / 1391 MB/s |
 +----------------------------+-------------------+-------------------+

I have no idea why ublk is generally slower when setup with 8 I/O queues. The
qd=32 4K random write with 8 jobs is generally faster with ublk than zloop, but
that varies. I tracked that down to CPU utilization which is generally much
better (all CPUs used) with ublk compared to zloop, as zloop is at the mercy of
the workqueue code and how it schedules unbound work items.

Also, I do not understand why the sequential write workload is so much slower
with ublk. That baffles me and I have no explanations.

With a faster PCIe gen4 16-core AMD Epyc-2 machine, I get:

Single queue:
=============
                              +-------------------+-------------------+
                              | ublk (IOPS / BW)  | zloop (IOPS / BW) |
 +----------------------------+-------------------+-------------------+
 | QD=1,    4K rnd wr, 1 job  | 6824 / 28.0 MB/s  | 7320 / 30.0 MB/s  |
 | QD=32,   4K rnd wr, 8 jobs | 50.9k / 208 MB/s  | 41.7k / 171 MB/s  |
 | QD=32, 128K rnd wr, 1 job  | 15.6k / 2046 MB/s | 18.5k / 2430 MB/s |
 | QD=32, 128K seq wr, 1 job  | 6237 / 818 MB/s   | 22.5k / 2943 MB/s |
 +----------------------------+-------------------+-------------------+

8 queues:
=========
                              +-------------------+-------------------+
                              | ublk (IOPS / BW)  | zloop (IOPS / BW) |
 +----------------------------+-------------------+-------------------+
 | QD=1,    4K rnd wr, 1 job  | 6884 / 28.2 MB/s  | 7707 / 31.6 MB/s  |
 | QD=32,   4K rnd wr, 8 jobs | 39.4k / 161 MB/s  | 46.8k / 192 MB/s  |
 | QD=32, 128K rnd wr, 1 job  | 12.2k / 1597 MB/s | 18.8k / 2460 MB/s |
 | QD=32, 128K seq wr, 1 job  | 6391 / 799 MB/s   | 21.4k / 2802 MB/s |
 +----------------------------+-------------------+-------------------+

The same pattern repeats again: ublk with 8 queues is slower, but it is faster
with a single queue for the qd=32 4K random write with 8 jobs case.

Simplicity need to be observed from multiple dimensions, 300 vs. 1500 LoC has
shown something already, IMO.

Sure. But given the very complicated syntax of rust, a lower LoC for rust
compared to C is very subjective in my opinion.

I said "simplicity" in the context of the driver use. And rublk is not as
simple to use as zloop as it needs rust/cargo installed which is not an
acceptable dependency for xfstests. Furthermore, it is very annoying to have to
change the nofile ulimit to allow rublk to open all the zone files for a large
disk (zloop does not need that)

-- 
Damien Le Moal
Western Digital Research