Re: [PATCH RFC] vfs: add a O_NOMTIME flag

On Wed, May 06, 2015 at 03:00:12PM -0700, Zach Brown wrote:

Add the O_NOMTIME flag which prevents mtime from being updated which can
greatly reduce the IO overhead of writes to allocated and initialized
regions of files.

Hmmm. How do backup programs now work out if the file has changed
and hence needs copying again? ie. applications using this will
break other critical infrastructure in subtle ways.

ceph servers can have loads where they perform O_DIRECT overwrites of
allocated file data and then sync to make sure that the O_DIRECT writes
are flushed from write caches.  If the writes dirty the inode with mtime
updates then the syncs also write out the metadata needed to track the
inodes which can add significant iop and latency overhead.

The ceph servers don't use mtime at all.  They're using the local file
system as a backing store and any backups would be driven by their upper
level ceph metadata.  For ceph, slow IO from mtime updates in the file
system is as daft as if we had block devices slowing down IO for
per-block write timestamps that file systems never use.

In simple tests a O_DIRECT|O_NOMTIME overwriting write followed by a
sync went from 2 serial write round trips to 1 in XFS and from 4 serial
IO round trips to 1 in ext4.

file_update_time() checks for O_NOMTIME and aborts the update if it's
set, just like the current check for the in-kernel inode flag
S_NOCMTIME.  I didn't update any other mtime update sites. They could be
added as we decide that it's appropriate to do so.

I opted not to name the flag O_NOCMTIME because I didn't want the name
to imply that ctime updates would be prevented for other inode changes
like updating i_size in truncate.  Not updating ctime is a side-effect
of removing mtime updates when it's the only thing changing in the
inode.

If adding this, wouldn't we want to unify O_NOMTIME and
FMODE_NOCMTIME at the same time?

i.e. it makes no sense to add O_NOMTIME and not add O_NOCMTIME,
likewise it makes no sense to have two different "no mtime"
detection mechanisms.  i.e. file_is_nomtime(file)) should return
true for both files opened with O_NOMTIME, files that have had
FMODE_NOCMTIME added to them and inodes with the S_NOCMTIME flag
set on them.

The criteria for using O_NOMTIME is the same as for using O_NOATIME:
owning the file or having the CAP_FOWNER capability.  If we're not
comfortable allowing owners to prevent mtime/ctime updates then we
should add a tunable to allow O_NOMTIME.  Maybe a mount option?

I dislike "turn off safety for performance" options because Joe
SpeedRacer will always select performance over safety.

Cheers,

Dave.
-- 
Dave Chinner
david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org

On Thu, May 07, 2015 at 10:26:17AM +1000, Dave Chinner wrote:

On Wed, May 06, 2015 at 03:00:12PM -0700, Zach Brown wrote:

quoted

Add the O_NOMTIME flag which prevents mtime from being updated which can
greatly reduce the IO overhead of writes to allocated and initialized
regions of files.

Hmmm. How do backup programs now work out if the file has changed
and hence needs copying again? ie. applications using this will
break other critical infrastructure in subtle ways.

By using backup infrastructure that doesn't use cmtime.  Like btrfs
send/recv.  Or application level backups that know how to do
incrementals from metadata in giant database files, say, without
walking, comparing, and copying the entire thing.

quoted

I opted not to name the flag O_NOCMTIME because I didn't want the name
to imply that ctime updates would be prevented for other inode changes
like updating i_size in truncate.  Not updating ctime is a side-effect
of removing mtime updates when it's the only thing changing in the
inode.

If adding this, wouldn't we want to unify O_NOMTIME and
FMODE_NOCMTIME at the same time?

I could see that, sure.

quoted

The criteria for using O_NOMTIME is the same as for using O_NOATIME:
owning the file or having the CAP_FOWNER capability.  If we're not
comfortable allowing owners to prevent mtime/ctime updates then we
should add a tunable to allow O_NOMTIME.  Maybe a mount option?

I dislike "turn off safety for performance" options because Joe
SpeedRacer will always select performance over safety.

Well, for ceph there's no safety concern.  They never use cmtime in
these files.

So are you suggesting not implementing this and making them rework their
IO paths to avoid the fs maintaining mtime so that we don't give Joe
Speedracer more rope?  Or are we talking about adding some speed bumps
that ceph can flip on that might give Joe Speedracer pause?

- z

quoted

quoted

The criteria for using O_NOMTIME is the same as for using O_NOATIME:
owning the file or having the CAP_FOWNER capability.  If we're not
comfortable allowing owners to prevent mtime/ctime updates then we
should add a tunable to allow O_NOMTIME.  Maybe a mount option?

I dislike "turn off safety for performance" options because Joe
SpeedRacer will always select performance over safety.

Well, for ceph there's no safety concern.  They never use cmtime in
these files.

So are you suggesting not implementing this and making them rework their
IO paths to avoid the fs maintaining mtime so that we don't give Joe
Speedracer more rope?  Or are we talking about adding some speed bumps
that ceph can flip on that might give Joe Speedracer pause?

Maybe one way to make it less of an attractive nuisance would be to hide
it under open_by_handle_at().  Like xfs_open_by_handle() does today but
we probably don't want to unconditionally add it to the generic path so
we'd have a flag.

They want to move to opening by handles anyway to avoid dirent lookups
when opening cold files.

- z

On Thu, 7 May 2015, Zach Brown wrote:

On Thu, May 07, 2015 at 10:26:17AM +1000, Dave Chinner wrote:

quoted

On Wed, May 06, 2015 at 03:00:12PM -0700, Zach Brown wrote:

quoted

The criteria for using O_NOMTIME is the same as for using O_NOATIME:
owning the file or having the CAP_FOWNER capability.  If we're not
comfortable allowing owners to prevent mtime/ctime updates then we
should add a tunable to allow O_NOMTIME.  Maybe a mount option?

I dislike "turn off safety for performance" options because Joe
SpeedRacer will always select performance over safety.

Well, for ceph there's no safety concern.  They never use cmtime in
these files.

So are you suggesting not implementing this and making them rework their
IO paths to avoid the fs maintaining mtime so that we don't give Joe
Speedracer more rope?  Or are we talking about adding some speed bumps
that ceph can flip on that might give Joe Speedracer pause?

I think this is the fundamental question: who do we give the ammunition 
to, the user or app writer, or the sysadmin?

One might argue that we gave the user a similar power with O_NOATIME (the 
power to break applications that assume atime is accurate).  Here we give 
developers/users the power to not update mtime and suffer the consequences 
(like, obviously, breaking mtime-based backups).  It should be pretty 
obvious to anyone using the flag what the consequences are.

Note that we can suffer similar lapses in mtime with fdatasync followed by 
a system crash.  And as Andy points out it's semi-broken for writable 
mmap.  The crash case is obviously a slightly different thing, but the 
idea that mtime can't always be trusted certainly isn't crazy talk.

Or, we can be conservative and require a mount option so that the admin 
has to explicitly allow behavior that might break some existing 
assumptions about mtime/ctime ('-o user_noatime' I guess?).

I'm happy either way, so long as in the end an unprivileged ceph daemon 
avoids the useless work.  In our case we always own the entire mount/disk, 
so a mount option is just fine.

Thanks!
sage

quoted

quoted

quoted

quoted

"Sage" == Sage Weil [off-list ref] writes:

Sage> On Thu, 7 May 2015, Zach Brown wrote:

quoted

On Thu, May 07, 2015 at 10:26:17AM +1000, Dave Chinner wrote:

quoted

On Wed, May 06, 2015 at 03:00:12PM -0700, Zach Brown wrote:

quoted

The criteria for using O_NOMTIME is the same as for using O_NOATIME:
owning the file or having the CAP_FOWNER capability.  If we're not
comfortable allowing owners to prevent mtime/ctime updates then we
should add a tunable to allow O_NOMTIME.  Maybe a mount option?

I dislike "turn off safety for performance" options because Joe
SpeedRacer will always select performance over safety.

Well, for ceph there's no safety concern.  They never use cmtime in
these files.

So are you suggesting not implementing this and making them rework their
IO paths to avoid the fs maintaining mtime so that we don't give Joe
Speedracer more rope?  Or are we talking about adding some speed bumps
that ceph can flip on that might give Joe Speedracer pause?

Sage> I think this is the fundamental question: who do we give the
Sage> ammunition to, the user or app writer, or the sysadmin?

Sage> One might argue that we gave the user a similar power with
Sage> O_NOATIME (the power to break applications that assume atime is
Sage> accurate).  Here we give developers/users the power to not
Sage> update mtime and suffer the consequences (like, obviously,
Sage> breaking mtime-based backups).  It should be pretty obvious to
Sage> anyone using the flag what the consequences are.

Not modifying atime doesn't really break anything except people who
think they can tell when a file was last accessed.  Which isn't
critical (unless your in a paranoid security conscious place...) but
MTIME is another beast entirely.   Turning that off is going to break
lots of hidden assumptions.  

Sage> Note that we can suffer similar lapses in mtime with fdatasync
Sage> followed by a system crash.  And as Andy points out it's
Sage> semi-broken for writable mmap.  The crash case is obviously a
Sage> slightly different thing, but the idea that mtime can't always
Sage> be trusted certainly isn't crazy talk.

True, but after a crash... people expect and understand there might be
corruption in a filesystem.  

Sage> Or, we can be conservative and require a mount option so that
Sage> the admin has to explicitly allow behavior that might break some
Sage> existing assumptions about mtime/ctime ('-o user_noatime' I
Sage> guess?).


Sage> I'm happy either way, so long as in the end an unprivileged ceph
Sage> daemon avoids the useless work.  In our case we always own the
Sage> entire mount/disk, so a mount option is just fine.

I agree with the mount option, makes it crystal clear.  And then it's
on the sysadmin/owner of the system to understand (ha!) the problems.

This is all me speaking with my Sysadmin hat firmly on my head.

Sage> I think this is the fundamental question: who do we give the
Sage> ammunition to, the user or app writer, or the sysadmin?

Sage> One might argue that we gave the user a similar power with
Sage> O_NOATIME (the power to break applications that assume atime is
Sage> accurate).  Here we give developers/users the power to not
Sage> update mtime and suffer the consequences (like, obviously,
Sage> breaking mtime-based backups).  It should be pretty obvious to
Sage> anyone using the flag what the consequences are.

Not modifying atime doesn't really break anything except people who
think they can tell when a file was last accessed.  Which isn't
critical (unless your in a paranoid security conscious place...) but
MTIME is another beast entirely.   Turning that off is going to break
lots of hidden assumptions.  

Sage> Note that we can suffer similar lapses in mtime with fdatasync
Sage> followed by a system crash.  And as Andy points out it's
Sage> semi-broken for writable mmap.  The crash case is obviously a
Sage> slightly different thing, but the idea that mtime can't always
Sage> be trusted certainly isn't crazy talk.

True, but after a crash... people expect and understand there might be
corruption in a filesystem.

Umm. No; people do not expect anything newer than ext3 to get
corrupted, ever.

In fact, I did not know about fdatasync/crash. That's rather nasty
surprise.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html