On Thu, Oct 28, 2021 at 11:24:08AM -0700, Ira Weiny wrote:

On Thu, Oct 28, 2021 at 01:52:27PM +0800, JeffleXu wrote:

quoted

On 10/27/21 10:36 PM, Vivek Goyal wrote:

quoted

[snip]

quoted

Is the biggest issue the lack of visibility to see if the device supports DAX?

Not necessarily. I think for me two biggest issues are.

- Should dax be enabled by default in server as well. If we do that,
  server will have to make extra ioctl() call on every LOOKUP and GETATTR
  fuse request. Local filesystems probably can easily query FS_XFLAGS_DAX
  state but doing extra syscall all the time will probably be some cost
  (No idea how much).

I tested the time cost from virtiofsd's perspective (time cost of
passthrough_ll.c:lo_do_lookup()):
- before per inode DAX feature: 2~4 us
- after per inode DAX feature: 7~8 us

It is within expectation, as the introduction of per inode DAX feature,
one extra ioctl() system call is introduced.

Also the time cost from client's perspective (time cost of
fs/fuse/dir.c:fuse_lookup_name())
- before per inode DAX feature: 25~30 us
- after per inode DAX feature: 30~35 us

That is, ~15%~20% performance loss.

Currently we do ioctl() to query the persitent inode flags every time
FUSE_LOOKUP request is received, maybe we could cache the result of
ioctl() on virtiofsd side, but I have no idea how to intercept the
runtime modification to these persistent indoe flags from other
processes on host, e.g. sysadmin on host, to maintain the cache consistency.

Do you really expect the dax flag to change on individual files a lot?  This in
itself is an expensive operation as the FS has to flush the inode.

No, we do not expect it to change often. But in a shared filesystem it
could be changed by somebody else. So we can't cache it in virtiofsd.
Even if we cache it we will need mechanism to invalidate cache if
some other client changed it. 

quoted

So if the default behavior of client side is 'dax=inode', and virtiofsd
disables per inode DAX by default (neither '-o dax=server|attr' is

I'm not following what dax=server or dax=attr is?

These are just the virtiofs daemon option names we are considering to
allow daemon to switch between different kind of policies. These names
are not final. As of now dax=attr is suggesting that look for FS_XFLAG_DAX
flag on inode and enable DAX on inode accordingly. dax=server means
that server can choose other policy to enable/disable DAX on an inode
(and can ignore FS_XFLAG_DAX). 

quoted

specified for virtiofsd) for the sake of performance, then guest won't
see DAX enabled and thus won't be surprised. This can reduce the
behavior change to the minimum.

What processes, other than virtiofsd have 'control' of these files?

Guest process  or user can change these flags. virtiofsd is not going
to modify this flag. It will just query this flag and respond to client
to enable DAX if this flag/attr is set on inode.

I know that a sysadmin could come in and change the dax flag but I think that
is like saying a sys-admin can come in and change your .bashrc and your
environment is suddenly different.  We have to trust the admins not to do stuff
like that.  So I don't think admins are going to be changing the dax flag on
files out from under 'users'; in this case virtiofsd.  Right?

Right. Generally I don't expect that on host anybody will change it. But I
will not rule it out because host is the one preparing initial filesystem
for the guest and if admin/tools on host want to set FS_XFLAG_DAX on
some of the inodes to begin with, so be it. Guest will boot with that
initial filesystem state.

That means that virtiofsd could cache the status and avoid the performance
issues above correct?

This directory could be shared also. That means multiple guests are
sharing same directory (each guest has one corresponding virtiofsd
instance running). That means if one guest changes the property of
one of the files, other guests/virtiofsd will have no idea that property
has changed.

Vivek

Ira

quoted

quoted

- So far if virtiofs is mounted without any of the dax options, just
  by looking at mount option, I could tell, DAX is not enabled on any
  of the files. But that will not be true anymore. Because dax=inode
  be default, it is possible that server upgrade enabled dax on some
  or all the files.

  I guess I will have to stick to same reason given by ext4/xfs. That is
  to determine whether DAX is enabled on a file or not, you need to
  query STATX_ATTR_DAX flag. That's the only way to conclude if DAX is
  being used on a file or not. Don't look at filesystem mount options
  and reach a conclusion (except the case of dax=never).

-- 
Thanks,
Jeffle