On Mon, 11 May 2015, Trond Myklebust wrote:
On Mon, May 11, 2015 at 12:39 PM, Sage Weil [off-list ref] wrote:
quoted
On Mon, 11 May 2015, Dave Chinner wrote:
quoted
On Sun, May 10, 2015 at 07:13:24PM -0400, Trond Myklebust wrote:
quoted
On Fri, May 8, 2015 at 6:24 PM, Sage Weil [off-list ref] wrote:
quoted
I'm sure you realize what we're try to achieve is the same "invisible IO"
that the XFS open by handle ioctls do by default. Would you be more
comfortable if this option where only available to the generic
open_by_handle syscall, and not to open(2)?
It should be an ioctl(). It has no business being part of
open_by_handle either, since that is another generic interface.
Our use-case doesn't make sense on network file systems, but it does on
any reasonably featureful local filesystem, and the goal is to be generic
there. If mtime is critical to a network file system's consistency it
seems pretty reasonable to disallow/ignore it for just that file system
(e.g., by masking off the flag at open time), as others won't have that
same problem (cephfs doesn't, for example).
Perhaps making each fs opt-in instead of handling it in a generic path
would alleviate this concern?
The issue isn't whether or not you have a network file system, it's
whether or not you want users to be able to manage data. mtime isn't
useful for the application (which knows whether or not it has changed
the file) or for the filesystem (ditto). It exists, rather, in order
to enable data management by users and other applications, letting
them know whether or not the data contents of the file have changed,
and when that change occurred.
Agreed.
If you are able to guarantee that your users don't care about that,
then fine, but that would be a very special case that doesn't fit the
way that most data centres are run. Backups are one case where mtime
matters, tiering and archiving is another.
This is true, although I argue it is becoming increasingly common for the
data management (including backups and so forth) to be layered not on top
of the POSIX file system but on something higher up in the stack. This is
true of pretty much any distributed system (ceph, cassandra, mongo, etc.,
and I assume commercial databases like Oracle, too) where backups,
replication, and any other DR strategies need to be orchestrated across
nodes to be consistent--simply copying files out from underneath them is
already insufficient and a recipe for disaster.
There is a growing category of applications that can benefit from this
capability...
Neither of these examples
cases are under the control of the application that calls
open(O_NOMTIME).
Wouldn't a mount option (e.g., allow_nomtime) address this concern? Only
nodes provisioned explicitly to run these systems would be enable this
option.
quoted
quoted
I'm happy for it to be an ioctl interface - even an XFS specific
interface if you want to go that route, Sage - and it probably
should emit a warning to syslog first time it is used so there is
trace for bug triage purposes. i.e. we know the app is not using
mtime updates, so bug reports that are the result of mtime
mishandling don't result in large amounts of wasted developer time
trying to understand them...
A warning on using the interface (or when mounting with user_nomtime)
sounds reasonable.
I'd rather not make this XFS specific as other local filesystmes (ext4,
f2fs, possibly btrfs) would similarly benefit. (And if we want to target
XFS specifically the existing XFS open-by-handle ioctl is sufficient as it
already does O_NOMTIME unconditionally.)
Lack of a namespace, doesn't imply that you don't want to manage the
data. The whole point of using object storage instead of plain old
block storage is to be able to provide whatever metadata you still
need in order to manage the object.
Yeah, agreed--this is presumably why open_by_handle(2) (which is what we'd
like to use) doesn't assume O_NOMTIME.
Thanks!
sage
On Mon, May 11, 2015 at 10:30:58AM -0700, Sage Weil wrote:
On Mon, 11 May 2015, Trond Myklebust wrote:
quoted
On Mon, May 11, 2015 at 12:39 PM, Sage Weil [off-list ref] wrote:
quoted
On Mon, 11 May 2015, Dave Chinner wrote:
quoted
On Sun, May 10, 2015 at 07:13:24PM -0400, Trond Myklebust wrote:
quoted
On Fri, May 8, 2015 at 6:24 PM, Sage Weil [off-list ref] wrote:
quoted
I'm sure you realize what we're try to achieve is the same "invisible IO"
that the XFS open by handle ioctls do by default. Would you be more
comfortable if this option where only available to the generic
open_by_handle syscall, and not to open(2)?
It should be an ioctl(). It has no business being part of
open_by_handle either, since that is another generic interface.
Our use-case doesn't make sense on network file systems, but it does on
any reasonably featureful local filesystem, and the goal is to be generic
there. If mtime is critical to a network file system's consistency it
seems pretty reasonable to disallow/ignore it for just that file system
(e.g., by masking off the flag at open time), as others won't have that
same problem (cephfs doesn't, for example).
Perhaps making each fs opt-in instead of handling it in a generic path
would alleviate this concern?
The issue isn't whether or not you have a network file system, it's
whether or not you want users to be able to manage data. mtime isn't
useful for the application (which knows whether or not it has changed
the file) or for the filesystem (ditto). It exists, rather, in order
to enable data management by users and other applications, letting
them know whether or not the data contents of the file have changed,
and when that change occurred.
Agreed.
quoted
If you are able to guarantee that your users don't care about that,
then fine, but that would be a very special case that doesn't fit the
way that most data centres are run. Backups are one case where mtime
matters, tiering and archiving is another.
This is true, although I argue it is becoming increasingly common for the
data management (including backups and so forth) to be layered not on top
of the POSIX file system but on something higher up in the stack. This is
In the cloud storage world, yes. In the rest of the world, no.
It's the rest of the world we are worried about here. :/
quoted
Neither of these examples
cases are under the control of the application that calls
open(O_NOMTIME).
Wouldn't a mount option (e.g., allow_nomtime) address this concern? Only
nodes provisioned explicitly to run these systems would be enable this
option.
Back to my Joe Speedracer comments.....
I'm not sure what the right answer is - mount options are simply too
easy to add without understanding the full implications of them.
e.g. we didn't merge FALLOC_FL_NO_HIDE_STALE simply because it was
too dangerous for unsuspecting users. This isn't at that same level
or concern, but it's still a landmine we want to avoid users from
arming without realising it...
quoted
quoted
quoted
I'm happy for it to be an ioctl interface - even an XFS specific
interface if you want to go that route, Sage - and it probably
should emit a warning to syslog first time it is used so there is
trace for bug triage purposes. i.e. we know the app is not using
mtime updates, so bug reports that are the result of mtime
mishandling don't result in large amounts of wasted developer time
trying to understand them...
A warning on using the interface (or when mounting with user_nomtime)
sounds reasonable.
I'd rather not make this XFS specific as other local filesystmes (ext4,
f2fs, possibly btrfs) would similarly benefit. (And if we want to target
XFS specifically the existing XFS open-by-handle ioctl is sufficient as it
already does O_NOMTIME unconditionally.)
Lack of a namespace, doesn't imply that you don't want to manage the
data. The whole point of using object storage instead of plain old
block storage is to be able to provide whatever metadata you still
need in order to manage the object.
Yeah, agreed--this is presumably why open_by_handle(2) (which is what we'd
like to use) doesn't assume O_NOMTIME.
Right - the XFS ioctls were designed specifically for applications
that interacted directly with the structure of XFS filesystems and
so needed invisible IO (e.g. online defragmenter). IOWs, they are
not interfaces intended for general usage. They are also only
available to root, so a typical user application won't be making use
of them, either.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
On Tue, 12 May 2015, Dave Chinner wrote:
quoted
quoted
Neither of these examples cases are under the control of the
application that calls open(O_NOMTIME).
Wouldn't a mount option (e.g., allow_nomtime) address this concern? Only
nodes provisioned explicitly to run these systems would be enable this
option.
Back to my Joe Speedracer comments.....
I'm not sure what the right answer is - mount options are simply too
easy to add without understanding the full implications of them.
e.g. we didn't merge FALLOC_FL_NO_HIDE_STALE simply because it was
too dangerous for unsuspecting users. This isn't at that same level
or concern, but it's still a landmine we want to avoid users from
arming without realising it...
quoted
quoted
quoted
quoted
I'm happy for it to be an ioctl interface - even an XFS specific
interface if you want to go that route, Sage - and it probably
should emit a warning to syslog first time it is used so there is
trace for bug triage purposes. i.e. we know the app is not using
mtime updates, so bug reports that are the result of mtime
mishandling don't result in large amounts of wasted developer time
trying to understand them...
A warning on using the interface (or when mounting with user_nomtime)
sounds reasonable.
I'd rather not make this XFS specific as other local filesystmes (ext4,
f2fs, possibly btrfs) would similarly benefit. (And if we want to target
XFS specifically the existing XFS open-by-handle ioctl is sufficient as it
already does O_NOMTIME unconditionally.)
Lack of a namespace, doesn't imply that you don't want to manage the
data. The whole point of using object storage instead of plain old
block storage is to be able to provide whatever metadata you still
need in order to manage the object.
Yeah, agreed--this is presumably why open_by_handle(2) (which is what we'd
like to use) doesn't assume O_NOMTIME.
Right - the XFS ioctls were designed specifically for applications
that interacted directly with the structure of XFS filesystems and
so needed invisible IO (e.g. online defragmenter). IOWs, they are
not interfaces intended for general usage. They are also only
available to root, so a typical user application won't be making use
of them, either.
I understand that's what they're intended for, but I'm having a hard time
parsing out the difference between what they *do* and what O_NOMTIME + -o
allow_nomtime does. The open-by-handle ioctls have nothing to do with the
online XFS format--they simply allow you to open a file via an opaque
handle (albeit a differently formatted one than the generic
open_by_handle_at(2)). They also force you into an O_NOMTIME-equivalent
mode.
AFAICS the only difference that I see is that
1) the ioctl is XFS specific. (As open_by_handle_at(2) demonstrates, this
needn't be the case.)
2) the NOMTIME mode is only available via the open-by-handle interface,
not open(2).
3) it is an ioctl interface, and thus more obscure. (Well, there is a
libhandle library, but it doesn't seem to be widely used.)
Would you object less if
1) the O_NOMTIME flag were only available via open_by_handle_at(2)?
2) an equivalent ioctl were implemented for each file system of interest
that (say) called into open_by_handle_at(2) code, adding in the O_NOMTIME
flag?
3) O_NOMTIME required root (vs a mount option that requires root and
unpriviledged O_NOMTIME)?
Just trying to tease apart which part is problematic...
Thanks!
sage
On Tue, May 12, 2015 at 04:12:46PM -0700, Sage Weil wrote:
On Tue, 12 May 2015, Dave Chinner wrote:
quoted
quoted
quoted
quoted
I'd rather not make this XFS specific as other local filesystmes (ext4,
f2fs, possibly btrfs) would similarly benefit. (And if we want to target
XFS specifically the existing XFS open-by-handle ioctl is sufficient as it
already does O_NOMTIME unconditionally.)
Lack of a namespace, doesn't imply that you don't want to manage the
data. The whole point of using object storage instead of plain old
block storage is to be able to provide whatever metadata you still
need in order to manage the object.
Yeah, agreed--this is presumably why open_by_handle(2) (which is what we'd
like to use) doesn't assume O_NOMTIME.
Right - the XFS ioctls were designed specifically for applications
that interacted directly with the structure of XFS filesystems and
so needed invisible IO (e.g. online defragmenter). IOWs, they are
not interfaces intended for general usage. They are also only
available to root, so a typical user application won't be making use
of them, either.
I understand that's what they're intended for, but I'm having a hard time
parsing out the difference between what they *do* and what O_NOMTIME + -o
allow_nomtime does. The open-by-handle ioctls have nothing to do with the
online XFS format--they simply allow you to open a file via an opaque
handle (albeit a differently formatted one than the generic
open_by_handle_at(2)). They also force you into an O_NOMTIME-equivalent
mode.
Actually, the handle is dervied from the information on disk. We
don't do directory lookups to build handles in many cases, we do a
bulkstat to get *on-disk* inode information (inode number, generation,
timestamps, etc) and then use that to build a handle in userspace
*and* validate the file has not changed since the infomration was
retrieved and the handle was built.
AFAICS the only difference that I see is that
1) the ioctl is XFS specific. (As open_by_handle_at(2) demonstrates, this
needn't be the case.)
Of course - it's been in use for 15 years longer than the generic
interface. :)
2) the NOMTIME mode is only available via the open-by-handle interface,
not open(2).
Right, because of the XFS handle interfaces are intended for
invisible IO which is required by applications interacting directly
with the XFS on-disk data layout.
3) it is an ioctl interface, and thus more obscure. (Well, there is a
libhandle library, but it doesn't seem to be widely used.)
The library only exists for xfsdump and the HSMs that interact
directly with the XFS on disk data. These are very constrained
applications.
Would you object less if
1) the O_NOMTIME flag were only available via open_by_handle_at(2)?
Which limits it to files that have already by created and written to
disk, otherwise there is no handle....
2) an equivalent ioctl were implemented for each file system of interest
that (say) called into open_by_handle_at(2) code, adding in the O_NOMTIME
flag?
Seems like a silly hoop to jump through. I was thinking of a
root-only fcntl() style flag that could be set, but....
3) O_NOMTIME required root (vs a mount option that requires root and
unpriviledged O_NOMTIME)?
Just trying to tease apart which part is problematic...
... it's very existence ias either a open or fcntl flag is still
problematic. :/
The concept of it being an on-disk attribute flag is less prone to
silent abuse - it's easily discoverable and is persistent. And it's
managable if we make it an "inherit from parent" style flag, because
then ceph can simply set it on the root dir, and every file it then
creates will not do mtime updates.
The other thing that is worth noting here is that we also have a
NODUMP flag on disk (chattr +d). Hence we could define that the
nomtime attribute also implies/sets the nodump attribute, and hence
makes it clear and upfront that turning on the nomtime inode
attribute will mean the files with this set will not get backed up
by mtime sensitive backup programs....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com