On Tue, Sep 29, 2020 at 6:54 AM Damien Le Moal [off-list ref] wrote:

On 2020/09/29 3:58, Kanchan Joshi wrote:
[...]

quoted

ZoneFS is better when it is about dealing at single-zone granularity,
and direct-block seems better when it is about grouping zones (in
various ways including striping). The latter case (i.e. grouping
zones) requires more involved mapping, and I agree that it can be left
to application (for both ZoneFS and raw-block backends).
But when an application tries that on ZoneFS, apart from mapping there
would be additional cost of indirection/fd-management (due to
file-on-files).

There is no indirection in zonefs. fd-to-struct file/inode conversion is very
fast and happens for every system call anyway, regardless of what the fd
represents. So I really do not understand what your worry is here.

file-over-files.....if application (user-space FS/DB) has to place
file-abstraction over zonefs again, to group/map the zones in a
different manner (larger files, striped mapping etc.).
Imagine a file over say 4 zones.....with zonefs backend, application
will invoke kernel at least 4 times to open the fds. With raw-block
backend, these system calls won't be required in the first place.

If you are
worried about overhead/performance, then please show numbers. If something is
wrong, we can work on fixing it.

quoted

And if new features (zone-append for now) are available only on
ZoneFS, it forces application to use something that maynot be most
optimal for its need.

"may" is not enough to convince me...

quoted

Coming to the original problem of plumbing append - I think divergence
started because RWF_APPEND did not have any meaning for block device.
Did I miss any other reason?

Correct.

quoted

How about write-anywhere semantics (RWF_RELAXED_WRITE or
RWF_ANONYMOUS_WRITE flag) on block-dev.

"write-anywhere" ? What do you mean ? That is not possible on zoned devices,
even with zone append, since you at least need to guarantee that zones have
enough unwritten space to accept an append command.

quoted

Zone-append works a lot like write-anywhere on block-dev (or on any
other file that combines multiple-zones, in non-sequential fashion).

That is an over-simplification that is not helpful at all. Zone append is not
"write anywhere" at all. And "write anywhere" is not a concept that exist on
regular block devices anyway. Writes only go to the offset that the user
decided, through lseek(), pwrite() or aio->aio_offset. It is not like the block
layer decides where the writes land. The same constraint applies to zone append:
the user decide the target zone. That is not "anywhere". Please be precise with
wording and implied/desired semantic. Narrow down the scope of your concept
names for clarity.

This -
<start>
Issue write on offset X with no flag, it happens at X.
Issue write on offset X with "anywhere" flag, it happens
anywhere....and application comes to know that on completion.
</start>
Above is fairly generic as far as high-level interface is concerned.
Return offset can be file-pointer or supplied-offset or something
completely different. If anywhere-flag is passed, the caller should
not assume anything and must explicitly check where write happened.
The "anywhere" part can be decided by the component that implements
the interface.
For zoned block dev,  this "anywhere" can come by issuing zone-append
underneath. Some other components are free to implement "anywhere" in
another way, or do nothing....in that case write continues to happen
at X.

For zoned raw-block, we have got an address-space that contains N
zones placed sequentially.
Write on offset O1 with anywhere flag: => The O1 is rounded-down to
zone-start (say Z1) by direct-io code, zone-append is issued on Z1,
and completion-offset O2 is returned.
write-anywhere on another offset O3 of address space => zone-start
could be Z2, and completion-offset O4 is returned.
Application picks completion offsets O3 and O4 and goes about its
business, not needing to know about Z1 or Z2.

And talking about "file that combines multiple-zones" would mean that we are now
back in FS land, not raw block device file accesses anymore. So which one are we
talking about ?

About user-space FS/DB/SDS using zoned-storage, aiming optimized placement.
In all this discussion, I have been referring to ZoneFS and Raw-block
as backends for higher-level abstraction residing in user-space.

It looks like you are confusing what the application does and
how it uses whatever usable interface to the device with what that interface
actually is. It is very confusing.

quoted

quoted

quoted

Also it seems difficult (compared to block dev) to fit simple-copy TP
in ZoneFS. The new
command needs: one NVMe drive, list of source LBAs and one destination
LBA. In ZoneFS, we would deal with N+1 file-descriptors (N source zone
file, and one destination zone file) for that. While with block
interface, we do not need  more than one file-descriptor representing
the entire device. With more zone-files, we face open/close overhead too.

Are you expecting simple-copy to allow requests that are not zone aligned ? I do
not think that will ever happen. Otherwise, the gotcha cases for it would be far
too numerous. Simple-copy is essentially an optimized regular write command.
Similarly to that command, it will not allow copies over zone boundaries and
will need the destination LBA to be aligned to the destination zone WP. I have
not checked the TP though and given the NVMe NDA, I will stop the discussion here.

TP is ratified, if that is the problem you are referring to.

Ah. Yes. Got confused with ZRWA. Simple-copy is a different story anyway. Let's
not mix it into zone append user interface please.

quoted

quoted

filesend() could be used as the interface for simple-copy. Implementing that in
zonefs would not be that hard. What is your plan for simple-copy interface for
raw block device ? An  ioctl ? filesend() too ? As as with any other user level
API, we should not be restricted to a particular device type if we can avoid it,
so in-kernel emulation of the feature is needed for devices that do not have
simple-copy or scsi extended copy. filesend() seems to me like the best choice
since all of that is already implemented there.

At this moment, ioctl as sync and io-uring for async. sendfile() and
copy_file_range() takes two fds....with that we can represent copy
from one source zone to another zone.
But it does not fit to represent larger copy (from N source zones to
one destination zone).

nvme passthrough ? If that does not fit your use case, then think of an
interface, its definition/semantic and propose it. But again, use a different
thread. This is mixing up zone-append and simple copy, which I do not think are
directly related.

quoted

Not sure if I am clear, perhaps sending RFC would be better for
discussion on simple-copy.

Separate this discussion from zone append please. Mixing up 2 problems in one
thread is not helpful to make progress.

Fine.


-- 
Joshi