On Fri, 30 Jul 2021, Qu Wenruo wrote:

On 2021/7/30 上午10:36, NeilBrown wrote:

quoted

I've been pondering all the excellent feedback, and what I have learnt
from examining the code in btrfs, and I have developed a different
perspective.

Great! Some new developers into the btrfs realm!

:-)

quoted

Maybe "subvol" is a poor choice of name because it conjures up
connections with the Volumes in LVM, and btrfs subvols are very different
things.  Btrfs subvols are really just subtrees that can be treated as a
unit for operations like "clone" or "destroy".

As such, they don't really deserve separate st_dev numbers.

Maybe the different st_dev numbers were introduced as a "cheap" way to
extend to size of the inode-number space.  Like many "cheap" things, it
has hidden costs.

Maybe objects in different subvols should still be given different inode
numbers.  This would be problematic on 32bit systems, but much less so on
64bit systems.

The patch below, which is just a proof-of-concept, changes btrfs to
report a uniform st_dev, and different (64bit) st_ino in different subvols.

It has problems:
  - it will break any 32bit readdir and 32bit stat.  I don't know how big
    a problem that is these days (ino_t in the kernel is "unsigned long",
    not "unsigned long long). That surprised me).
  - It might break some user-space expectations.  One thing I have learnt
    is not to make any assumption about what other people might expect.

Wouldn't any filesystem boundary check fail to stop at subvolume boundary?

You mean like "du -x"?? Yes.  You would lose the misleading illusion
that there are multiple filesystems.  That is one user-expectation that
would need to be addressed before people opt-in

Then it will go through the full btrfs subvolumes/snapshots, which can
be super slow.

quoted

However, it would be quite easy to make this opt-in (or opt-out) with a
mount option, so that people who need the current inode numbers and will
accept the current breakage can keep working.

I think this approach would be a net-win for NFS export, whether BTRFS
supports it directly or not.  I might post a patch which modifies NFS to
intuit improved inode numbers for btrfs exports....

Some extra ideas, but not familiar with VFS enough to be sure.

Can we generate "fake" superblock for each subvolume?

I don't see how that would help.  Either subvols are like filesystems
and appear in /proc/mounts, or they aren't like filesystems and don't
get different st_dev.  Either of these outcomes can be achieved without
fake superblocks.  If you really need BTRFS subvols to have some
properties of filesystems but not all, then you are in for a whole world
of pain.

Maybe btrfs subvols should be treated more like XFS "managed trees".  At
least there you have precedent and someone else to share the pain.
Maybe we should train people to use "quota" to check the usage of a
subvol, rather than "du" (which will stop working with my patch if it
contains refs to other subvols) or "df" (which already doesn't work), or
"btrs df"

Like using the subolume UUID to replace the FSID of each subvolume.
Could that migrate the problem?

Which problem, exactly?  My first approach to making subvols work on NFS
took essentially that approach.  It was seen (quite reasonably) as a
hack to work around poor behaviour in btrfs.

Given that NFS has always seen all of a btrfs filesystem as have a
uniform fsid, I'm now of the opinion that we don't want to change that,
but should just fix the duplicate-inode-number problem.

If I could think of some way for NFSD to see different inode numbers
than VFS, I would push hard for fixs nfsd by giving it more sane inode
numbers.

Thanks,
NeilBrown