On Fri, 18 Jun 2021, Wang Yugui wrote:

Hi,

quoted

On Thu, 17 Jun 2021, Wang Yugui wrote:

quoted

quoted

Can we go back to the beginning.  What, exactly, is the problem you are
trying to solve?  How can you demonstrate the problem?

NeilBrown

I nfs/exported a btrfs with 2 subvols and 2 snapshot(subvol).

# btrfs subvolume list /mnt/test
ID 256 gen 53 top level 5 path sub1
ID 260 gen 56 top level 5 path sub2
ID 261 gen 57 top level 5 path .snapshot/sub1-s1
ID 262 gen 57 top level 5 path .snapshot/sub2-s1

and then mount.nfs4 it to /nfs/test.

# /bin/find /nfs/test/
/nfs/test/
find: File system loop detected; '/nfs/test/sub1' is part of the same file system loop as '/nfs/test/'.
/nfs/test/.snapshot
find: File system loop detected; '/nfs/test/.snapshot/sub1-s1' is part of the same file system loop as '/nfs/test/'.
find: File system loop detected; '/nfs/test/.snapshot/sub2-s1' is part of the same file system loop as '/nfs/test/'.
/nfs/test/dir1
/nfs/test/dir1/a.txt
find: File system loop detected; '/nfs/test/sub2' is part of the same file system loop as '/nfs/test/'

/bin/find report 'File system loop detected'. so I though there is
something wrong.

Certainly something is wrong.  The error message implies that some
directory is reporting the same dev an ino as an ancestor directory.
Presumably /nfs/test and /nfs/test/sub1.
Can you confirm that please. e.g. run the command

   stat /nfs/test /nfs/test/sub1
and examine the output.

# stat /nfs/test /nfs/test/sub1
  File: /nfs/test
  Size: 42              Blocks: 32         IO Block: 32768  directory
Device: 36h/54d Inode: 256         Links: 1
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-06-18 13:50:55.409457648 +0800
Modify: 2021-06-13 10:05:10.830825901 +0800
Change: 2021-06-13 10:05:10.830825901 +0800
 Birth: -
  File: /nfs/test/sub1
  Size: 8               Blocks: 0          IO Block: 32768  directory
Device: 36h/54d Inode: 256         Links: 1
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-06-18 13:51:14.463621411 +0800
Modify: 2021-06-12 21:59:10.598089917 +0800
Change: 2021-06-12 21:59:10.598089917 +0800
 Birth: -

same 'Device/Inode' are reported.


but the local btrfs mount,
# stat /mnt/test/ /mnt/test/sub1
  File: /mnt/test/
  Size: 42              Blocks: 32         IO Block: 4096   directory
Device: 33h/51d Inode: 256         Links: 1
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-06-18 13:50:55.409457648 +0800
Modify: 2021-06-13 10:05:10.830825901 +0800
Change: 2021-06-13 10:05:10.830825901 +0800
 Birth: -
  File: /mnt/test/sub1
  Size: 8               Blocks: 0          IO Block: 4096   directory
Device: 34h/52d Inode: 256         Links: 1
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-06-18 13:51:14.463621411 +0800
Modify: 2021-06-12 21:59:10.598089917 +0800
Change: 2021-06-12 21:59:10.598089917 +0800
 Birth: -

'stat' command should cause nfs/crossmnt to happen auto, and then return
the 'stat' result?

quoted

As sub1 is considered a different file system, it should have a
different dev number.  NFS will assign a different device number only
when the server reports a different fsid.  The Linux NFS server will
report a different fsid if d_mountpoint() is 'true' for the dentry, and
follow_down() results in no change the the vfsmnt,dentry in a 'struct
path'.

You have already said that d_mountpoint doesn't work for btrfs, so that
is part of the problem.  NFSD doesn't trust d_mountpoint completely as
it only reports that the dentry is a mountpoint in some namespace, not
necessarily in this namespace.  So you really need to fix
nfsd_mountpoint.

I suggest you try adding your "dirty fix" to nfsd_mountpoint() so that
it reports the root of a btrfs subvol as a mountpoint, and see if that
fixes the problem.  It should change the problem at least.  You would
need to get nfsd_mountpoint() to return '1' in this case, not '2'.

NeilBrown

I changed the return value from 2 to 1.
        if (nfsd4_is_junction(dentry))
                return 1;
+       if (is_btrfs_subvol_d(dentry))
+               return 1;
        if (d_mountpoint(dentry))

but the crossmnt still does not happen auto.

I tried to mount the subvol manual, 
# mount.nfs4 T7610:/mnt/test/sub1 /nfs/test/sub1
mount.nfs4: Stale file handle

we add trace to is_btrfs_subvol_d(), it works as expected.
+static inline bool is_btrfs_subvol_d(const struct dentry *dentry)
+{
+    bool ret= dentry->d_inode && dentry->d_inode->i_ino == 256ULL &&
+		dentry->d_sb && dentry->d_sb->s_magic == 0x9123683E;
+	printk(KERN_INFO "is_btrfs_subvol_d(%s)=%d\n", dentry->d_name.name, ret);
+	return ret;
+}

It seems more fixes are needed.

I think the problem is that the submount doesn't appear in /proc/mounts.
"nfsd_fh()" in nfs-utils needs to be able to map from the uuid for a
filesystem to the mount point.  To do this it walks through /proc/mounts
checking the uuid of each filesystem.  If a filesystem isn't listed
there, it obviously fails.

I guess you could add code to nfs-utils to do whatever "btrfs subvol
list" does to make up for the fact that btrfs doesn't register in
/proc/mounts.

NeilBrown