From: Shaohua Li <hidden> Date: 2011-01-20 03:21:49
On Thu, 2011-01-20 at 11:05 +0800, Andrew Morton wrote:
On Thu, 20 Jan 2011 10:48:33 +0800 Shaohua Li [off-list ref] wrote:
quoted
On Thu, 2011-01-20 at 10:42 +0800, Andrew Morton wrote:
quoted
On Thu, 20 Jan 2011 10:30:47 +0800 Shaohua Li [off-list ref] wrote:
quoted
quoted
I don't know if this is worth addressing. Perhaps require that the
filp refers to the root of the fs?
I didn't see why this is needed, but I can limit the fip to the root of
the fs.
I don't think it matters much either. The only problem I can see is if
we were to later try to extend the ioctl into a per-file thing.
since we return page range, a metadata page might be shared by several
files, which makes the per-file thing doesn't work. For a fs using
trees, it's even more hard to distinguish a file's metadata
hm, why. A query for "which blocks need to be read to access this
file" may return blocks which are shared with other files, but it's
still useful info. Because it will represent vastly less data (and
hence IO) than the current fs-wide thing.
ok
Now I actually look at it, I cannot find any documentation for the ioctl!
I'll write more doc in later post.
It seems to return a single offset/length tuple which refers to the
btrfs metadata "file", with the intent that this tuple later be fed
into a btrfs-specific readahead ioctl.
I can see how this might be used with say fatfs or ext3 where all
metadata resides within the blockdev address_space. But how is a
filesytem which keeps its metadata in multiple address_spaces supposed
to use this interface?
Oh, this looks like a big problem, thanks for letting me know such
filesystems. is it possible specific filesystem mapping multiple
address_space ranges to a virtual big ranges? the new ioctls handle the
mapping.
If the issue can't be solved, we can only add the metadata readahead for
specific implementation like my initial post instead of a generic
interface.
Thanks,
Shaohua
From: Andrew Morton <hidden> Date: 2011-01-20 04:10:14
On Thu, 20 Jan 2011 11:21:49 +0800 Shaohua Li [off-list ref] wrote:
quoted
It seems to return a single offset/length tuple which refers to the
btrfs metadata "file", with the intent that this tuple later be fed
into a btrfs-specific readahead ioctl.
I can see how this might be used with say fatfs or ext3 where all
metadata resides within the blockdev address_space. But how is a
filesytem which keeps its metadata in multiple address_spaces supposed
to use this interface?
Oh, this looks like a big problem, thanks for letting me know such
filesystems. is it possible specific filesystem mapping multiple
address_space ranges to a virtual big ranges? the new ioctls handle the
mapping.
I'm not sure what you mean by that.
ext2, minix and probably others create an address_space for each
directory. Heaven knows what xfs does (for example).
If the issue can't be solved, we can only add the metadata readahead for
specific implementation like my initial post instead of a generic
interface.
Well. One approach would be for the kernel to report the names of all
presently-cached files. And for each file, report the offsets of all
the pages which are presently in pagecache. This all gets put into a
database.
At cold-boot time we open all those files and read the relevant files.
To optimise that further, userspace would need to use fibmap to work
out the LBA(s) of each page, and then read the pages in an optimised order.
To optimise that even further, userspace would need to find the on-disk
locations all the metadata for each file, generate the metadata->data
dependencies and then incorporate that into the reading order.
I actually wrote code to do all this. Gad, it was ten years ago. I
forget how it works, but I do recall that it pioneered the technology
of doing (effecticely) a sys_write(1, ...) from a kernel module, so the
module's output appears on modprobe's stdout and can be redirected to
another file or a pipe. So sue me! It's in
http://userweb.kernel.org/~akpm/stuff/fboot.tar.gz. Good luck with
that ;)
<looks>
It walked mem_map[], indentifying pagecache pages, walking back from
the page* all the way to the filename then logging the pathname and the
file's pagecache indexes. It also handled the blockdev superblock,
where all the ext3 metadata resides.
There are much smarter ways of doing this of course, especially with
the vfs data structures which we later added.
<googles>
According to http://kerneltrap.org/node/2157 it sped up cold boot by
"10%", whatever that means. Seems that I wasn't sufficiently impressed
by that and got distracted.
I'm not sure any of that was very useful, really. A full-on coldboot
optimiser really wants visibility into every disk block which need to
be read, and then mechanisms to tell the kernel to load those blocks
into the correct address_spaces. That's hard, because file data
depends on file metadata. A vast simplification would be to do it in
two disk passes: read all the metadata on pass 1 then all the data on
pass 2.
A totally different approach is to reorder all the data and metadata
on-disk, so no special cold-boot processing is needed at all.
And a third approach is to save all the cache into a special
file/partition/etc and to preload all that into kernel data structures
at boot. Obviously this one is ricky/tricky because the on-disk
replica of the real data can get out of sync with the real data.
From: Dave Chinner <david@fromorbit.com> Date: 2011-01-20 04:41:27
On Wed, Jan 19, 2011 at 08:10:14PM -0800, Andrew Morton wrote:
On Thu, 20 Jan 2011 11:21:49 +0800 Shaohua Li [off-list ref] wrote:
quoted
quoted
It seems to return a single offset/length tuple which refers to the
btrfs metadata "file", with the intent that this tuple later be fed
into a btrfs-specific readahead ioctl.
I can see how this might be used with say fatfs or ext3 where all
metadata resides within the blockdev address_space. But how is a
filesytem which keeps its metadata in multiple address_spaces supposed
to use this interface?
Oh, this looks like a big problem, thanks for letting me know such
filesystems. is it possible specific filesystem mapping multiple
address_space ranges to a virtual big ranges? the new ioctls handle the
mapping.
I'm not sure what you mean by that.
ext2, minix and probably others create an address_space for each
directory. Heaven knows what xfs does (for example).
In 2.6.39 it won't even use address spaces for metadata caching.
Besides, XFS already has pretty sophisticated metadata readahead
built in - it's one of the reasons why the XFS directory code scales
so well on cold cache lookups of arge directories - so I don't see
much need for such an interface for XFS.
Perhaps btrfs would be better served by implementing speculative
metadata readahead in the places where it makes sense (e.g. readdir)
bcause it will improve cold-cache performance on a much wider range
of workloads than at just boot-time....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
From: Shaohua Li <hidden> Date: 2011-01-20 05:44:57
On Thu, 2011-01-20 at 12:41 +0800, Dave Chinner wrote:
On Wed, Jan 19, 2011 at 08:10:14PM -0800, Andrew Morton wrote:
quoted
On Thu, 20 Jan 2011 11:21:49 +0800 Shaohua Li [off-list ref] wrote:
quoted
quoted
It seems to return a single offset/length tuple which refers to the
btrfs metadata "file", with the intent that this tuple later be fed
into a btrfs-specific readahead ioctl.
I can see how this might be used with say fatfs or ext3 where all
metadata resides within the blockdev address_space. But how is a
filesytem which keeps its metadata in multiple address_spaces supposed
to use this interface?
Oh, this looks like a big problem, thanks for letting me know such
filesystems. is it possible specific filesystem mapping multiple
address_space ranges to a virtual big ranges? the new ioctls handle the
mapping.
I'm not sure what you mean by that.
ext2, minix and probably others create an address_space for each
directory. Heaven knows what xfs does (for example).
In 2.6.39 it won't even use address spaces for metadata caching.
Besides, XFS already has pretty sophisticated metadata readahead
built in - it's one of the reasons why the XFS directory code scales
so well on cold cache lookups of arge directories - so I don't see
much need for such an interface for XFS.
Perhaps btrfs would be better served by implementing speculative
metadata readahead in the places where it makes sense (e.g. readdir)
bcause it will improve cold-cache performance on a much wider range
of workloads than at just boot-time....
I don't know about xfs. A sophisticated metadata readahead might make
metadata async, but I thought it's impossible it can removes the disk
seek. Since metadata and data usually lives in different disk block
ranges, doing data readahead will unavoidable read metadata and cause
disk seek between reading data and metadata.
Thanks,
Shaohua
On Thu, Jan 20, 2011 at 01:44:57PM +0800, Li, Shaohua wrote:
On Thu, 2011-01-20 at 12:41 +0800, Dave Chinner wrote:
quoted
On Wed, Jan 19, 2011 at 08:10:14PM -0800, Andrew Morton wrote:
quoted
On Thu, 20 Jan 2011 11:21:49 +0800 Shaohua Li [off-list ref] wrote:
quoted
quoted
It seems to return a single offset/length tuple which refers to the
btrfs metadata "file", with the intent that this tuple later be fed
into a btrfs-specific readahead ioctl.
I can see how this might be used with say fatfs or ext3 where all
metadata resides within the blockdev address_space. But how is a
filesytem which keeps its metadata in multiple address_spaces supposed
to use this interface?
Oh, this looks like a big problem, thanks for letting me know such
filesystems. is it possible specific filesystem mapping multiple
address_space ranges to a virtual big ranges? the new ioctls handle the
mapping.
I'm not sure what you mean by that.
ext2, minix and probably others create an address_space for each
directory. Heaven knows what xfs does (for example).
In 2.6.39 it won't even use address spaces for metadata caching.
Besides, XFS already has pretty sophisticated metadata readahead
built in - it's one of the reasons why the XFS directory code scales
so well on cold cache lookups of arge directories - so I don't see
much need for such an interface for XFS.
Perhaps btrfs would be better served by implementing speculative
metadata readahead in the places where it makes sense (e.g. readdir)
bcause it will improve cold-cache performance on a much wider range
of workloads than at just boot-time....
I don't know about xfs. A sophisticated metadata readahead might make
metadata async, but I thought it's impossible it can removes the disk
seek. Since metadata and data usually lives in different disk block
ranges, doing data readahead will unavoidable read metadata and cause
disk seek between reading data and metadata.
It's standard practice to do in-kernel heuristic readahead for large
directories. It's irrelevant to data/metadata interleaving.
It's exactly interleaved reads that makes readahead a must-have.
Think about interleavingly reading 2+ large files :)
Thanks,
Fengguang
From: Dave Chinner <hidden> Date: 2011-01-24 04:29:59
On Thu, Jan 20, 2011 at 01:44:57PM +0800, Shaohua Li wrote:
On Thu, 2011-01-20 at 12:41 +0800, Dave Chinner wrote:
quoted
On Wed, Jan 19, 2011 at 08:10:14PM -0800, Andrew Morton wrote:
quoted
On Thu, 20 Jan 2011 11:21:49 +0800 Shaohua Li [off-list ref] wrote:
quoted
quoted
It seems to return a single offset/length tuple which refers to the
btrfs metadata "file", with the intent that this tuple later be fed
into a btrfs-specific readahead ioctl.
I can see how this might be used with say fatfs or ext3 where all
metadata resides within the blockdev address_space. But how is a
filesytem which keeps its metadata in multiple address_spaces supposed
to use this interface?
Oh, this looks like a big problem, thanks for letting me know such
filesystems. is it possible specific filesystem mapping multiple
address_space ranges to a virtual big ranges? the new ioctls handle the
mapping.
I'm not sure what you mean by that.
ext2, minix and probably others create an address_space for each
directory. Heaven knows what xfs does (for example).
In 2.6.39 it won't even use address spaces for metadata caching.
Besides, XFS already has pretty sophisticated metadata readahead
built in - it's one of the reasons why the XFS directory code scales
so well on cold cache lookups of arge directories - so I don't see
much need for such an interface for XFS.
Perhaps btrfs would be better served by implementing speculative
metadata readahead in the places where it makes sense (e.g. readdir)
bcause it will improve cold-cache performance on a much wider range
of workloads than at just boot-time....
I don't know about xfs. A sophisticated metadata readahead might make
metadata async, but I thought it's impossible it can removes the disk
seek.
Nothing you do will remove the disk seek. What readahead is supposed
to do is _minimise the latency_ of the disk seek.
Since metadata and data usually lives in different disk block
ranges, doing data readahead will unavoidable read metadata and cause
disk seek between reading data and metadata.
Which comes back to how well the filesystem lays out the metadata
related to the data that needs to be read. In the case of XFS, the
metadata it needs is already in the inode, so once the inodes are
read into memory, there is no extra metadata seeks between data
seeks.
That is, if you are using XFS all you need to do in terms of
metadata readahead is stat every file needed by the boot process.
The optimal order for doing this is simply by ordering them in
ascending inode number. IOWs, the problem can be optimised without
any special kernel interfaces to do metadata readahead, especially
if you multithread the stat() walk to avoid blocking on IO that
metadata readahead hasn't already brought into cache....
IIRC, btrfs tends to keep all it's per-inode metadata close together
like XFS does, so it should be read at the same time the inode is
read.
Indeed, the dependencies of readahead are pretty well understood. A
demonstration of optimising reading of file data across a complex
directory heirarchy is well deomonstrated by this little tool from
Chris Mason:
http://oss.oracle.com/~mason/acp/
I suspect that applying such a technique to the problem of optimising
boot-time IO pattern with net you the same gains as this new kernel
API will. And it will do it in a manner that is filesystem
agnostic...
Cheers,
Dave.
--
Dave Chinner
david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org