[PATCH 0/3] Extended file stat system call [ver #2]

Implement a new system call to provide an extended and further extensible stat
function.

=======
CHANGES
=======

 (*) Added an optional auxiliary data buffer pointer argument to sys_statxat()
     and moved some of the stuff into it.

 (*) Discarded "has an ACL info flag" for now.

 (*) AFS doesn't need to set time granularity itself as the default is 1s
     anyway.


===========
DESCRIPTION
===========

The second of the associated patches is the main patch that provides this new
system call:

	long ret = statxat(int dfd,
			   const char *filename,
			   unsigned atflag,
			   unsigned mask,
			   struct statx *buffer,
			   struct statx_auxinfo *auxinfo);

which is more fully documented in the main patch's description.

This new stat function provides a number of useful features, in summary:

 (1) More information: creation time, data version number, flags/attributes.
     A subset of these is available through a number of filesystems (such as
     CIFS, NFS, AFS, Ext4 and BTRFS).

 (2) Lightweight stat: Ask for just those details of interest, and allow a
     netfs (such as NFS) to approximate anything not of interest, possibly
     without going to the server.

 (3) Heavyweight stat: Force a netfs to go to the server, even if it thinks its
     cached attributes are up to date.

 (4) Allow the filesystem to indicate what it can/cannot provide: A filesystem
     can now say it doesn't support a standard stat feature if that isn't
     available.

 (5) Make the fields a consistent size on all arches, and make them large.

 (6) Can be extended by using more request flags and appending further data
     after the end of the standard return data.

Note that no lstat() equivalent is required as that can be implemented through
statxat() with atflag == 0.  There is also no fstat() equivalent as that can be
implemented through statxat() with filename == NULL and the relevant fd passed
as dfd.


=======
PATCHES
=======

Patch 1 introduces IOC flags for Windows Hidden, Archive and System attributes.

Patch 2 defines the statxat() system call.

Patch 3 implements extended stat facilities for AFS.  Other fs patches to go
through their respective trees once the core is in.


==============
CONSIDERATIONS
==============

Should the default for a network fs be to do an unconditional (heavyweight)
stat with a flag to suppress going to the server to update the locally held
attributes and flushing pending writebacks?

Is it useful to pass the volume ID out?  Or should statfs() be used for this?

Is there anything else I can usefully add at the moment?


==========
TO BE DONE
==========

Autofs, ntfs, btrfs, ...

I should perhaps use u8/u32/u64 rather than uint8/32/64_t.

Handle remote filesystems being offline and indicate this with
STATX_INFO_OFFLINE.

Should a "light stat" be explicitly requested by its own AT_ flag?


=======
TESTING
=======

There's a test program attached to the description for the main patch.  It can
be run as follows:
[root@andromeda tmp]# ./xstat -R /mnt/foo

	xstat(/mnt/foo) = 0
	0000: 000081a40000ffef 0000000000000001 0000020000000000 0000100000080000
	0020: 0000000000000000 0000000600000008 000000004f88499a 0000000136fd9208
	0040: 000000004f88499a 0000000136fd9208 000000004f8849b9 0000000106daf187
	0060: 000000004f8849b9 0000000106daf187 000000000000000c 000000000000000f
	0080: 0000000000000008 00000000484ebbef 0000000000000025 5949ebd4711efd82
	00a0: d3250b5c15d5e380 0000000000000000 0000000000000000 0000000000000000
	00c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
	00e0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
	results=ffef
	  Size: 15              Blocks: 8          IO Block: 4096    regular file
	Device: 08:06           Inode: 12          Links: 1    
	Access: (0644/-rw-r--r--)  Uid: 0   
	Gid: 0
	Access: 2012-04-13 16:43:22.922587656+0100
	Modify: 2012-04-13 16:43:53.115011975+0100
	Change: 2012-04-13 16:43:53.115011975+0100
	Create: 2012-04-13 16:43:22.922587656+0100
	Inode version: 484ebbefh
	Data version: 25h
	Inode flags: 00080000 (-------- ----e--- -------- --------)
	Information: 00000200 (-------- -------- ------a- --------)
	Volume ID: 82fd1e71d4eb4959-80e3d5155c0b25d3

David
---
David Howells (3):
      statxat: Provide IOC flags for Windows fs attributes
      statxat: Add a system call to make extended file stats available
      statxat: AFS: Return enhanced basic attributes


 arch/x86/ia32/sys_ia32.c         |    2 
 arch/x86/syscalls/syscall_32.tbl |    1 
 arch/x86/syscalls/syscall_64.tbl |    1 
 fs/afs/inode.c                   |   46 ++++-
 fs/ceph/inode.c                  |    2 
 fs/cifs/inode.c                  |    5 -
 fs/compat.c                      |    2 
 fs/nfsd/nfsxdr.c                 |    2 
 fs/stat.c                        |  348 ++++++++++++++++++++++++++++++++++++--
 include/linux/fs.h               |    3 
 include/linux/stat.h             |   16 ++
 include/linux/syscalls.h         |    6 +
 include/uapi/linux/fcntl.h       |    1 
 include/uapi/linux/fs.h          |    7 +
 include/uapi/linux/stat.h        |  164 ++++++++++++++++++
 15 files changed, 567 insertions(+), 39 deletions(-)

Provide IOC flags for Windows fs attributes so that they can be retrieved (or
even altered) using the FS_IOC_[GS]ETFLAGS ioctl and read using statxat().

Signed-off-by: David Howells <dhowells@redhat.com>
---

 include/uapi/linux/fs.h |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index 6c28b61bb690..8b966f48dd41 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h

@@ -191,10 +191,13 @@ struct inodes_stat_t {
 #define FS_EXTENT_FL			0x00080000 /* Extents */
 #define FS_DIRECTIO_FL			0x00100000 /* Use direct i/o */
 #define FS_NOCOW_FL			0x00800000 /* Do not cow file */
+#define FS_HIDDEN_FL			0x10000000 /* Windows hidden file attribute */
+#define FS_SYSTEM_FL			0x20000000 /* Windows system file attribute */
+#define FS_ARCHIVE_FL			0x40000000 /* Windows archive file attribute */
 #define FS_RESERVED_FL			0x80000000 /* reserved for ext2 lib */
 
-#define FS_FL_USER_VISIBLE		0x0003DFFF /* User visible flags */
-#define FS_FL_USER_MODIFIABLE		0x000380FF /* User modifiable flags */
+#define FS_FL_USER_VISIBLE		0x7003DFFF /* User visible flags */
+#define FS_FL_USER_MODIFIABLE		0x700380FF /* User modifiable flags */
 
 
 #define SYNC_FILE_RANGE_WAIT_BEFORE	1

Return enhanced basic attributes from the AFS filesystem.  This includes the
following:

 (1) The data version number as st_data_version.

 (2) STATX_INFO_AUTOMOUNT will be set on automount directories by virtue of
     S_AUTOMOUNT being set on the inode.  These are referrals to other volumes
     or other cells.

 (3) STATX_INFO_AUTODIR on a directory that does cell lookup for non-existent
     names and mounts them (typically mounted on /afs with -o autocell).  The
     resulting directories are marked STATX_INFO_FABRICATED as they do not
     actually exist in the mounted AFS directory.

 (4) Files, directories and symlinks accessed over AFS are marked
     STATX_INFO_REMOTE.  Local fake directories are marked
     STATX_INFO_FABRICATED.

 (5) STATX_INFO_NONSYSTEM_OWNERSHIP is set as the UID and GID retrieved from an
     AFS share may not be applicable on the system.

and also some auxiliary data:

 (6) The volume ID and type.

 (7) The volume name.

 (8) The cell name.

Signed-off-by: David Howells <dhowells@redhat.com>
---

 fs/afs/inode.c |   46 ++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 38 insertions(+), 8 deletions(-)

diff --git a/fs/afs/inode.c b/fs/afs/inode.c
index ce25d755b7aa..5789f68abbea 100644
--- a/fs/afs/inode.c
+++ b/fs/afs/inode.c

@@ -71,9 +71,9 @@ static int afs_inode_map_status(struct afs_vnode *vnode, struct key *key)
 	inode->i_uid		= vnode->status.owner;
 	inode->i_gid		= GLOBAL_ROOT_GID;
 	inode->i_size		= vnode->status.size;
-	inode->i_ctime.tv_sec	= vnode->status.mtime_server;
-	inode->i_ctime.tv_nsec	= 0;
-	inode->i_atime		= inode->i_mtime = inode->i_ctime;
+	inode->i_mtime.tv_sec	= vnode->status.mtime_server;
+	inode->i_mtime.tv_nsec	= 0;
+	inode->i_atime		= inode->i_ctime = inode->i_mtime;
 	inode->i_blocks		= 0;
 	inode->i_generation	= vnode->fid.unique;
 	inode->i_version	= vnode->status.data_version;

@@ -374,16 +374,46 @@ error_unlock:
 /*
  * read the attributes of an inode
  */
-int afs_getattr(struct vfsmount *mnt, struct dentry *dentry,
-		      struct kstat *stat)
+int afs_getattr(struct vfsmount *mnt, struct dentry *dentry, struct kstat *stat)
 {
-	struct inode *inode;
-
-	inode = dentry->d_inode;
+	struct inode *inode = dentry->d_inode;
 
 	_enter("{ ino=%lu v=%u }", inode->i_ino, inode->i_generation);
 
 	generic_fillattr(inode, stat);
+
+	stat->result_mask &= ~(STATX_ATIME | STATX_CTIME | STATX_BLOCKS);
+	stat->result_mask |= STATX_VERSION;
+	stat->version = inode->i_version;
+
+	if (test_bit(AFS_VNODE_AUTOCELL, &AFS_FS_I(inode)->flags))
+		stat->information |= STATX_INFO_AUTODIR;
+
+	if (test_bit(AFS_VNODE_PSEUDODIR, &AFS_FS_I(inode)->flags))
+		stat->information |= STATX_INFO_FABRICATED;
+	else
+		stat->information |= STATX_INFO_REMOTE;
+
+	stat->information |= STATX_INFO_NONSYSTEM_OWNERSHIP;
+
+	if (stat->auxinfo) {
+		struct afs_super_info *as = inode->i_sb->s_fs_info;
+		struct statx_auxinfo *aux = stat->auxinfo;
+
+		aux->sx_fsid = as->volume->vid;
+		/* construct a volume ID from the AFS volume ID and type */
+		aux->sx_volume_id[0] = as->volume->vid >> 24;
+		aux->sx_volume_id[1] = as->volume->vid >> 16;
+		aux->sx_volume_id[2] = as->volume->vid >> 8;
+		aux->sx_volume_id[3] = as->volume->vid >> 0;
+		aux->sx_volume_id[4] = as->volume->type;
+
+		strcpy(aux->sx_volume_name, as->volume->vlocation->vldb.name);
+		strcpy(aux->sx_domain_name, as->volume->cell->name);
+
+		aux->sx_mask = STATX_FSID | STATX_VOLUME_ID |
+			STATX_VOLUME_NAME | STATX_DOMAIN_NAME;
+	}
 	return 0;
 }
 

On 11/12/2013 01:41 PM, David Howells wrote:

Implement a new system call to provide an extended and further extensible stat
function.

=======
CHANGES
=======

 (*) Added an optional auxiliary data buffer pointer argument to sys_statxat()
     and moved some of the stuff into it.

 (*) Discarded "has an ACL info flag" for now.

 (*) AFS doesn't need to set time granularity itself as the default is 1s
     anyway.

David,

This looks like great work. I admire the thoroughness and
the detail of the patches. I don't have much more to say
other than it looks robust and very well put together.

You have CC'd the glibc development list on these changes
and I appreciate that as a heads up on new syscalls is very
very useful for the developers.

Do you have any other expectations from the glibc development
community? Are you looking for help wiring this up or will
you be posting glibc patches at a later date?

Cheers,
Carlos.

On Tue, Nov 12, 2013 at 06:41:35PM +0000, David Howells wrote:

Provide IOC flags for Windows fs attributes so that they can be retrieved (or
even altered) using the FS_IOC_[GS]ETFLAGS ioctl and read using statxat().

If you're adding FAT specific flags here into the generic xstat
interface, can we get a bunch of common XFS specific flags into this
interface as well so people don't need to use xfs_io to get/set
them?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com