On Thursday 11 August 2005 20:49, David Howells wrote:

Daniel Phillips [off-list ref] wrote:

quoted

To be honest I'm having some trouble following this through logically. 
I'll read through a few more times and see if that fixes the problem. 
This seems cluster-related, so I have an interest.

Well, perhaps I can explain the function for which I'm using this page flag
more clearly. You'll have to excuse me if it's covering stuff you don't
know, but I want to take it from first principles; plus this stuff might
well find its way into the kernel docs.


We want to use a relatively fast medium (such as RAM or local disk) to
speed up repeated accesses to a relatively slow medium (such as NFS, NBD,
CDROM) by means of caching the results of previous accesses to the slow
medium on the fast medium.

Now we already do this at one level: RAM. The page cache _is_ such a cache,
but whilst it's much faster than a disk, it is severely restricted in size

Did you just suggest that 16 TB/address_space is too small to cache NFS pages?

compared to media such as disks, it's more expensive

It is?

and it's contents generally don't last over power failure or reboots.

When used by RAMFS maybe.  But fortunately the page cache has a backing store 
API, in fact, that is its raison d'etre.

The major attribute of the page cache is that the CPU can access it
directly. 

You seem to have forgotten about non-resident pages.

So we want to add another level: local disk. The FS-Cache/CacheFS patches
permit such as AFS and NFS to use local disk as a cache.

The page cache already lets you do that.  I have not yet discerned a 
fundamental reason why you need to interface to another filesystem to 
implement backing store for an address_space.

So, assume that NFS is using a local disk cache (it doesn't matter whether
it's CacheFS, CacheFiles, or something else), and assume a process has a
file open through NFS.

The process attempts to read from the file. This causes the NFS readpage()
or readpages() operation to be invoked to load the data into the page cache
so that the CPU can make use of it.

So the NFS page reading algorithm first consults the disk cache.  Assume 
this returns a negative response - NFS will then read from the server into
the page cache. Under cacheless operation, it would then unlock the page
and the kernel could then let userspace play with it, but we're dealing
with a cache, and so the newly fetched data must be stored in the disk
cache for future retrieval.

NFS now has three choices:

 (1) It could institigate a write to the disk cache and wait for that to
     complete before unlocking the page and letting userspace see it, but
we don't know how long that might take.

Pages are typically unlocked while being written to backing store, e.g.:

http://lxr.linux.no/source/fs/buffer.c#L1839

What makes NFS special in this regard?

     CacheFS immediately dispatches a write BIO to get it DMA'd to the disk
as soon as possible, but something like CacheFiles is dependent on an
underlying filesystem - be it EXT3, ReiserFS, XFS, etc. - to perform the
write, and we've no control over that.

That is a problem you are in the process of inventing.

	Time to unlock: CacheMiss + NetRead + CacheWrite
	Cache reliable: Yes

 (2) It could just unlock the page and let userspace scribble on it whilst
     simultaneously writing it to the cache. But that means the DMA to the
     disk may pick up some of userspace's scribblings, and that means you
     can't trust what's in the cache in the event of a power loss.

I thought I saw a journal in there.  Anyway, if the user has asked for a racy 
write, that is what they should get.

     This can be alleviated by marking untrustworthy files in the cache,
but that then extends the management time in several ways.

	Time to unlock: CacheMiss + NetRead
	Cache reliable: No

I think your definition of trustworthy goes beyond what is required by Posix 
or Linux local filesystem semantics.

 (3) It could tell the cache that the page needs writing to disk and then
     unlock it for userspace to read, but intercept the change of a PTE
     pointing to this page when it loses its write protection (PTEs start
off read-only, generating a write protection fault on the first write).

We need to do something like this to implemented cross-node caching of 
shared-writeable mmaps.  This is another reason that your ideas need clear 
explanations: we need to go the rest of the way and get this sorted out for 
cluster filesystems in general, not just NFS (v4).  It does help a lot that 
you are attempting to explain what the needs of NFS actually are.  
Unfortunately, it seems you are proposing that this mechanism is essential 
even for single-node use, which is far from clear.

     The interceptor would then force userspace to wait for the cache to
     finish DMA'ing the page before writing to it.

     Similarly, the write() or prepare_write() operations would wait for
the cache to finish with that page.

Here you return to the assumption that the VFS should enforce per-page write 
granularity.  There is no such rule as far as I know.

	Time to unlock: CacheMiss + NetRead
	Cache reliable: Yes

I originally chose option (1), but then I saw just how much it affected
performance and worked on option (3).

I discarded option (2) because I want to be able to have some surety about
the state in the cache - I don't want to have to reinitialise it after a
power failure. Imagine if you cache /usr... Imagine if everyone in a very
large office caches /usr...


So, the way I implemented (3) is to use an extra page flag to indicate a
write underway to the cache, and thus allow cache write status to be
determined when someone wants to scribble on a page.

The fscache_write_page() function takes a pointer to a callback function.
In NFS this function clears the PG_fs_misc bit on the appropriate pages and
wakes up anyone who was waiting for this event (end_page_fs_misc()).

The NFS page_mkwrite() VMA op calls wait_on_page_fs_misc() to wait on that
page bit if it is set.

quoted

Who is using this interface?

AFS and NFS will both use it. There may be others eventually who use it for
the same purpose. CacheFS has a different use for it internally.

Let's try to clear up the page write atomicity question, please.  It seems 
your argument depends on it.

Regards,

Daniel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>