[BUG] fscache writing but not reading

While testing the fscache performance fixes [1] that were merged into 6.4-rc1
it appears that the caching no longer works. The client will write to the cache
but never reads.

I suspect this is related to known issue #1. However, I tested the client
with rsize less than, equal to, and greater than readahead, and in all cases
I see the issue.

If I apply both the patches [2], [3] from the known issues to 6.4-rc1 then the
cache works as expected. I suspect only patch [2] is required but have not
tested patch [2] without [3].

Testing
=======
For the test I was just using dd to read 300 x 1gb files from an NFS
share to fill the cache, then repeating the read.

In the first test run, /var/cache/fscache steadily filled until reaching
300 GB. The read I/O was less than 1 MB/s, and the write speed was fairly
constant 270 MB/s.

In the second run, /var/cache/fscache remained at 300 GB, so no new data was
being written. However, the read I/O remained at less than 1 MB/s and the write
rate at 270 MB/s.

    /var/cache/fscache
                | 1st run     | 2nd run
    disk usage  | 0 -> 300 GB | 300 GB
    read speed  | < 1 MB/s    | < 1 MB/s
    write speed | 270 MB/s    | 270 MB/s

This seems to imply that the already cached data was being read from the source
server and re-written to the cache.

Known Issues
============
1. Unit test setting rsize < readahead does not properly read from
fscache but re-reads data from the NFS server
* This will be fixed with another dhowells patch [2]:
  "[PATCH v6 2/2] mm, netfs, fscache: Stop read optimisation when
folio removed from pagecache"

2. "Cache volume key already in use" after xfstest runs involving
multiple mounts
* Simple reproducer requires just two mounts as follows:
 mount -overs=4.1,fsc,nosharecache -o
context=system_u:object_r:root_t:s0  nfs-server:/exp1 /mnt1
 mount -overs=4.1,fsc,nosharecache -o
context=system_u:object_r:root_t:s0  nfs-server:/exp2 /mnt2
* This should be fixed with dhowells patch [3]:
  "[PATCH v5] vfs, security: Fix automount superblock LSM init
problem, preventing NFS sb sharing"

References
==========

[1] https://lore.kernel.org/linux-nfs/20230220134308.1193219-1-dwysocha@redhat.com/
[2] https://lore.kernel.org/linux-nfs/20230216150701.3654894-1-dhowells@redhat.com/T/#mf3807fa68fb6d495b87dde0d76b5237833a0cc81
[3] https://lore.kernel.org/linux-kernel/217595.1662033775@warthog.procyon.org.uk/

On Tue, May 16, 2023 at 11:42 AM Chris Chilvers [off-list ref] wrote:

While testing the fscache performance fixes [1] that were merged into 6.4-rc1
it appears that the caching no longer works. The client will write to the cache
but never reads.

Thanks for the report.

If you reboot do you see reads from the cache?

You can check if the cache is being read from by looking in
/proc/fs/fscache/stats
at the "IO" line:
# grep IO /proc/fs/fscache/stats
IO     : rd=80030 wr=0

You might consider saving that file before your test, then another copy after,
and doing a diff:
# diff -u /tmp/fscachestats.1 /tmp/fscachestats.2

--- /tmp/fscachestats.1 2023-05-16 14:48:43.126158403 -0400
+++ /tmp/fscachestats.2 2023-05-16 14:54:05.421402702 -0400

@@ -1,14 +1,14 @@
 FS-Cache statistics
-Cookies: n=0 v=1 vcol=0 voom=0
-Acquire: n=0 ok=0 oom=0
-LRU    : n=0 exp=0 rmv=0 drp=0 at=0
+Cookies: n=5 v=1 vcol=0 voom=0
+Acquire: n=5 ok=5 oom=0
+LRU    : n=0 exp=5 rmv=0 drp=0 at=0
 Invals : n=0
-Updates: n=0 rsz=0 rsn=0
+Updates: n=5 rsz=0 rsn=0
 Relinqs: n=0 rtr=0 drop=0
 NoSpace: nwr=0 ncr=0 cull=6
-IO     : rd=0 wr=0
-RdHelp : RA=0 RP=0 WB=0 WBZ=0 rr=0 sr=0
+IO     : rd=40015 wr=0
+RdHelp : RA=40015 RP=0 WB=0 WBZ=0 rr=0 sr=0
 RdHelp : ZR=0 sh=0 sk=0
 RdHelp : DL=0 ds=0 df=0 di=0
-RdHelp : RD=0 rs=0 rf=0
+RdHelp : RD=40015 rs=40015 rf=0
 RdHelp : WR=0 ws=0 wf=0

I suspect this is related to known issue #1. However, I tested the client
with rsize less than, equal to, and greater than readahead, and in all cases
I see the issue.

If I apply both the patches [2], [3] from the known issues to 6.4-rc1 then the
cache works as expected. I suspect only patch [2] is required but have not
tested patch [2] without [3].

Agree it's likely only the patches from issue #1 are needed.
Let me ping dhowells and willy on that thread for issue #1 as it looks
stalled.

Testing
=======
For the test I was just using dd to read 300 x 1gb files from an NFS
share to fill the cache, then repeating the read.

Can you share:
1. NFS server you're using (is it localhost or something else)
2. NFS version

In the first test run, /var/cache/fscache steadily filled until reaching
300 GB. The read I/O was less than 1 MB/s, and the write speed was fairly
constant 270 MB/s.

In the second run, /var/cache/fscache remained at 300 GB, so no new data was
being written. However, the read I/O remained at less than 1 MB/s and the write
rate at 270 MB/s.

    /var/cache/fscache
                | 1st run     | 2nd run
    disk usage  | 0 -> 300 GB | 300 GB
    read speed  | < 1 MB/s    | < 1 MB/s
    write speed | 270 MB/s    | 270 MB/s

This seems to imply that the already cached data was being read from the source
server and re-written to the cache.

In addition to checking the above for the reads from the cache, you can also
see whether NFS reads are going over the wire pretty easily with a similar
technique.

Copy /proc/self/mounstats to a file before your test, then make a second copy
after the test, then run mountstats as follows:
mountstats -S /tmp/mountstats.1 -f /tmp/mountstats.2

Known Issues
============
1. Unit test setting rsize < readahead does not properly read from
fscache but re-reads data from the NFS server
* This will be fixed with another dhowells patch [2]:
  "[PATCH v6 2/2] mm, netfs, fscache: Stop read optimisation when
folio removed from pagecache"

2. "Cache volume key already in use" after xfstest runs involving
multiple mounts
* Simple reproducer requires just two mounts as follows:
 mount -overs=4.1,fsc,nosharecache -o
context=system_u:object_r:root_t:s0  nfs-server:/exp1 /mnt1
 mount -overs=4.1,fsc,nosharecache -o
context=system_u:object_r:root_t:s0  nfs-server:/exp2 /mnt2
* This should be fixed with dhowells patch [3]:
  "[PATCH v5] vfs, security: Fix automount superblock LSM init
problem, preventing NFS sb sharing"

References
==========

[1] https://lore.kernel.org/linux-nfs/20230220134308.1193219-1-dwysocha@redhat.com/
[2] https://lore.kernel.org/linux-nfs/20230216150701.3654894-1-dhowells@redhat.com/T/#mf3807fa68fb6d495b87dde0d76b5237833a0cc81
[3] https://lore.kernel.org/linux-kernel/217595.1662033775@warthog.procyon.org.uk/

--
Linux-cachefs mailing list
Linux-cachefs@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-cachefs

On Tue, 16 May 2023 at 20:28, David Wysochanski [off-list ref] wrote:

On Tue, May 16, 2023 at 11:42 AM Chris Chilvers [off-list ref] wrote:

quoted

While testing the fscache performance fixes [1] that were merged into 6.4-rc1
it appears that the caching no longer works. The client will write to the cache
but never reads.

Thanks for the report.

If you reboot do you see reads from the cache?

On the first read after a reboot it uses the cache, but subsequent
reads do not use the cache.

You can check if the cache is being read from by looking in
/proc/fs/fscache/stats
at the "IO" line:
# grep IO /proc/fs/fscache/stats
IO     : rd=80030 wr=0

Running the tests 4 times (twice before reboot, and twice after) give
the following metrics:
FS-Cache I/O (delta)
Run       rd        wr
 1         0    39,250
 2       130    38,894
 3    39,000         0
 4        72    38,991

Can you share:
1. NFS server you're using (is it localhost or something else)
2. NFS version

The NFS server and client are separate VMs on the same network.
Server NFS version: Ubuntu 22.04 jammy, kernel 5.15.0-1021-gcp
Client NFS version: Ubuntu 22.04 jammy, kernel 6.4.0-060400rc1-generic
(https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc1/)
Client nfs-utils: 2.6.3-rc6
Client cachefilesd: 0.10.10-0.2ubuntu1

In addition to checking the above for the reads from the cache, you can also
see whether NFS reads are going over the wire pretty easily with a similar
technique.

Copy /proc/self/mounstats to a file before your test, then make a second copy
after the test, then run mountstats as follows:
mountstats -S /tmp/mountstats.1 -f /tmp/mountstats.2

app read    = applications read bytes via read(2)
client read = client read bytes via NFS READ

Run           app read        client read
 1     322,122,547,200    322,122,547,200
 2     322,122,547,200    321,048,805,376
 3     322,122,547,200                  0
 4     322,122,547,200    321,593,053,184

I've put the full data in a GitHub gist, along with a graph collected
from a metrics agent:
https://gist.github.com/chilversc/54eb76155ad37b66cb85186e7449beaa
https://gist.githubusercontent.com/chilversc/54eb76155ad37b66cb85186e7449beaa/raw/09828c596d0cfc44bc0eb67f40e4033db202326e/metrics.png

Chris Chilvers [off-list ref] wrote:

While testing the fscache performance fixes [1] that were merged into 6.4-rc1
it appears that the caching no longer works. The client will write to the cache
but never reads.

Can you try reading from afs?  You would need to enable CONFIG_AFS_FS in your
kernel if it's not already set.

Install kafs-client and do:

	systemctl enable afs.mount
	md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
	cat /proc/fs/fscache/stats
	umount /afs/openafs.org
	md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
	cat /proc/fs/fscache/stats

David

On Thu, May 18, 2023 at 4:21 PM Chris Chilvers [off-list ref] wrote:

On Tue, 16 May 2023 at 20:28, David Wysochanski [off-list ref] wrote:

quoted

On Tue, May 16, 2023 at 11:42 AM Chris Chilvers [off-list ref] wrote:

quoted

While testing the fscache performance fixes [1] that were merged into 6.4-rc1
it appears that the caching no longer works. The client will write to the cache
but never reads.

Thanks for the report.

If you reboot do you see reads from the cache?

On the first read after a reboot it uses the cache, but subsequent
reads do not use the cache.

quoted

You can check if the cache is being read from by looking in
/proc/fs/fscache/stats
at the "IO" line:
# grep IO /proc/fs/fscache/stats
IO     : rd=80030 wr=0

Running the tests 4 times (twice before reboot, and twice after) give
the following metrics:
FS-Cache I/O (delta)
Run       rd        wr
 1         0    39,250
 2       130    38,894
 3    39,000         0
 4        72    38,991

quoted

Can you share:
1. NFS server you're using (is it localhost or something else)
2. NFS version

The NFS server and client are separate VMs on the same network.
Server NFS version: Ubuntu 22.04 jammy, kernel 5.15.0-1021-gcp
Client NFS version: Ubuntu 22.04 jammy, kernel 6.4.0-060400rc1-generic
(https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc1/)
Client nfs-utils: 2.6.3-rc6
Client cachefilesd: 0.10.10-0.2ubuntu1

quoted

In addition to checking the above for the reads from the cache, you can also
see whether NFS reads are going over the wire pretty easily with a similar
technique.

Copy /proc/self/mounstats to a file before your test, then make a second copy
after the test, then run mountstats as follows:
mountstats -S /tmp/mountstats.1 -f /tmp/mountstats.2

app read    = applications read bytes via read(2)
client read = client read bytes via NFS READ

Run           app read        client read
 1     322,122,547,200    322,122,547,200
 2     322,122,547,200    321,048,805,376
 3     322,122,547,200                  0
 4     322,122,547,200    321,593,053,184

I've put the full data in a GitHub gist, along with a graph collected
from a metrics agent:
https://gist.github.com/chilversc/54eb76155ad37b66cb85186e7449beaa
https://gist.githubusercontent.com/chilversc/54eb76155ad37b66cb85186e7449beaa/raw/09828c596d0cfc44bc0eb67f40e4033db202326e/metrics.png

Thanks Chris for all this info.  I see you're using NFSv3 so I'll
focus on that, and review all this info for clues.
I also have been working on some updated test cases and see some very
unusual behavior like you're reporting.

I also confirmed that adding the two patches for "Issue #1"  onto
6.4-rc1 resolve _most_ of the caching issues.
However, even after those patches, in some limited instances, there
are still NFS reads over the wire when there should only be reads from
the cache.
There may be multiple bugs here.

On Fri, 19 May 2023 at 10:40, David Howells [off-list ref] wrote:

Chris Chilvers [off-list ref] wrote:

quoted

While testing the fscache performance fixes [1] that were merged into 6.4-rc1
it appears that the caching no longer works. The client will write to the cache
but never reads.

Can you try reading from afs?  You would need to enable CONFIG_AFS_FS in your
kernel if it's not already set.

Install kafs-client and do:

        systemctl enable afs.mount
        md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
        cat /proc/fs/fscache/stats

FS-Cache statistics
Cookies: n=13 v=7 vcol=0 voom=0
Acquire: n=13 ok=13 oom=0
LRU    : n=1 exp=0 rmv=0 drp=0 at=2833
Invals : n=0
Updates: n=0 rsz=0 rsn=0
Relinqs: n=0 rtr=0 drop=0
NoSpace: nwr=0 ncr=0 cull=0
IO     : rd=0 wr=18
RdHelp : RA=18 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=1 sh=0 sk=0
RdHelp : DL=18 ds=18 df=0 di=0
RdHelp : RD=0 rs=0 rf=0
RdHelp : WR=18 ws=18 wf=0

This was on an instance that was only just created, so the cache was
initially unused (all the counters were 0).

        umount /afs/openafs.org
        md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
        cat /proc/fs/fscache/stats

FS-Cache statistics
Cookies: n=13 v=7 vcol=0 voom=0
Acquire: n=26 ok=26 oom=0
LRU    : n=1 exp=1 rmv=0 drp=0 at=467
Invals : n=0
Updates: n=0 rsz=0 rsn=0
Relinqs: n=13 rtr=0 drop=13
NoSpace: nwr=0 ncr=0 cull=0
IO     : rd=18 wr=18
RdHelp : RA=36 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=1 sh=0 sk=0
RdHelp : DL=18 ds=18 df=0 di=0
RdHelp : RD=18 rs=18 rf=0
RdHelp : WR=18 ws=18 wf=0

Looks like the cache is working fine with AFS. The second md5sum
seemed a lot quicker than the first.

On Tue, May 23, 2023 at 7:18 AM Chris Chilvers [off-list ref] wrote:

On Fri, 19 May 2023 at 10:40, David Howells [off-list ref] wrote:

quoted

Chris Chilvers [off-list ref] wrote:

quoted

While testing the fscache performance fixes [1] that were merged into 6.4-rc1
it appears that the caching no longer works. The client will write to the cache
but never reads.

Can you try reading from afs?  You would need to enable CONFIG_AFS_FS in your
kernel if it's not already set.

Install kafs-client and do:

        systemctl enable afs.mount
        md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
        cat /proc/fs/fscache/stats

FS-Cache statistics
Cookies: n=13 v=7 vcol=0 voom=0
Acquire: n=13 ok=13 oom=0
LRU    : n=1 exp=0 rmv=0 drp=0 at=2833
Invals : n=0
Updates: n=0 rsz=0 rsn=0
Relinqs: n=0 rtr=0 drop=0
NoSpace: nwr=0 ncr=0 cull=0
IO     : rd=0 wr=18
RdHelp : RA=18 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=1 sh=0 sk=0
RdHelp : DL=18 ds=18 df=0 di=0
RdHelp : RD=0 rs=0 rf=0
RdHelp : WR=18 ws=18 wf=0

This was on an instance that was only just created, so the cache was
initially unused (all the counters were 0).

quoted

        umount /afs/openafs.org
        md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
        cat /proc/fs/fscache/stats

FS-Cache statistics
Cookies: n=13 v=7 vcol=0 voom=0
Acquire: n=26 ok=26 oom=0
LRU    : n=1 exp=1 rmv=0 drp=0 at=467
Invals : n=0
Updates: n=0 rsz=0 rsn=0
Relinqs: n=13 rtr=0 drop=13
NoSpace: nwr=0 ncr=0 cull=0
IO     : rd=18 wr=18
RdHelp : RA=36 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=1 sh=0 sk=0
RdHelp : DL=18 ds=18 df=0 di=0
RdHelp : RD=18 rs=18 rf=0
RdHelp : WR=18 ws=18 wf=0

Looks like the cache is working fine with AFS. The second md5sum
seemed a lot quicker than the first.

--
Linux-cachefs mailing list
Linux-cachefs@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-cachefs

I don't think this test demonstrates the failure Chris is seeing, or
correct me if I'm wrong Chris.

Chris, does this same test fail for you with NFS on 6.4-rc1?  The same
test passes for me with NFSv3.
1st time through: writes to the cache seen, no reads: IO     : rd=0 wr=16
2nd time through: reads from the cache seen, no writes: IO     : rd=16 wr=16


# mount -o vers=3,fsc 127.0.0.1:/export/ /mnt/nfs
# cat /proc/fs/fscache/stats
FS-Cache statistics
Cookies: n=0 v=1 vcol=0 voom=0
Acquire: n=0 ok=0 oom=0
LRU    : n=0 exp=0 rmv=0 drp=0 at=0
Invals : n=0
Updates: n=0 rsz=0 rsn=0
Relinqs: n=0 rtr=0 drop=0
NoSpace: nwr=0 ncr=0 cull=0
IO     : rd=0 wr=0
RdHelp : RA=0 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=0 sh=0 sk=0
RdHelp : DL=0 ds=0 df=0 di=0
RdHelp : RD=0 rs=0 rf=0
RdHelp : WR=0 ws=0 wf=0
# md5sum /mnt/nfs/openafs-1.9.1-doc.tar.bz2
ca25530ec01afc3b71bc081bc93b87cf  /mnt/nfs/openafs-1.9.1-doc.tar.bz2
# cat /proc/fs/fscache/stats
FS-Cache statistics
Cookies: n=1 v=1 vcol=0 voom=0
Acquire: n=1 ok=1 oom=0
LRU    : n=1 exp=0 rmv=0 drp=0 at=6444
Invals : n=0
Updates: n=1 rsz=0 rsn=0
Relinqs: n=0 rtr=0 drop=0
NoSpace: nwr=0 ncr=0 cull=0
IO     : rd=0 wr=16
RdHelp : RA=16 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=1 sh=0 sk=0
RdHelp : DL=16 ds=16 df=0 di=0
RdHelp : RD=0 rs=0 rf=0
RdHelp : WR=16 ws=16 wf=0
# umount /mnt/nfs
# mount -o vers=3,fsc 127.0.0.1:/export/ /mnt/nfs
# md5sum /mnt/nfs/openafs-1.9.1-doc.tar.bz2
ca25530ec01afc3b71bc081bc93b87cf  /mnt/nfs/openafs-1.9.1-doc.tar.bz2
# cat /proc/fs/fscache/stats
FS-Cache statistics
Cookies: n=1 v=1 vcol=0 voom=0
Acquire: n=2 ok=2 oom=0
LRU    : n=1 exp=1 rmv=0 drp=0 at=6710
Invals : n=0
Updates: n=2 rsz=0 rsn=0
Relinqs: n=1 rtr=0 drop=1
NoSpace: nwr=0 ncr=0 cull=0
IO     : rd=16 wr=16
RdHelp : RA=32 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=1 sh=0 sk=0
RdHelp : DL=16 ds=16 df=0 di=0
RdHelp : RD=16 rs=16 rf=0
RdHelp : WR=16 ws=16 wf=0
# uname -r
6.4.0-rc1

On Fri, May 19, 2023 at 7:53 AM David Wysochanski [off-list ref] wrote:

On Thu, May 18, 2023 at 4:21 PM Chris Chilvers [off-list ref] wrote:

quoted

On Tue, 16 May 2023 at 20:28, David Wysochanski [off-list ref] wrote:

quoted

On Tue, May 16, 2023 at 11:42 AM Chris Chilvers [off-list ref] wrote:

quoted

While testing the fscache performance fixes [1] that were merged into 6.4-rc1
it appears that the caching no longer works. The client will write to the cache
but never reads.

Thanks for the report.

If you reboot do you see reads from the cache?

On the first read after a reboot it uses the cache, but subsequent
reads do not use the cache.

quoted

You can check if the cache is being read from by looking in
/proc/fs/fscache/stats
at the "IO" line:
# grep IO /proc/fs/fscache/stats
IO     : rd=80030 wr=0

Running the tests 4 times (twice before reboot, and twice after) give
the following metrics:
FS-Cache I/O (delta)
Run       rd        wr
 1         0    39,250
 2       130    38,894
 3    39,000         0
 4        72    38,991

quoted

Can you share:
1. NFS server you're using (is it localhost or something else)
2. NFS version

The NFS server and client are separate VMs on the same network.
Server NFS version: Ubuntu 22.04 jammy, kernel 5.15.0-1021-gcp
Client NFS version: Ubuntu 22.04 jammy, kernel 6.4.0-060400rc1-generic
(https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc1/)
Client nfs-utils: 2.6.3-rc6
Client cachefilesd: 0.10.10-0.2ubuntu1

quoted

In addition to checking the above for the reads from the cache, you can also
see whether NFS reads are going over the wire pretty easily with a similar
technique.

Copy /proc/self/mounstats to a file before your test, then make a second copy
after the test, then run mountstats as follows:
mountstats -S /tmp/mountstats.1 -f /tmp/mountstats.2

app read    = applications read bytes via read(2)
client read = client read bytes via NFS READ

Run           app read        client read
 1     322,122,547,200    322,122,547,200
 2     322,122,547,200    321,048,805,376
 3     322,122,547,200                  0
 4     322,122,547,200    321,593,053,184

I've put the full data in a GitHub gist, along with a graph collected
from a metrics agent:
https://gist.github.com/chilversc/54eb76155ad37b66cb85186e7449beaa
https://gist.githubusercontent.com/chilversc/54eb76155ad37b66cb85186e7449beaa/raw/09828c596d0cfc44bc0eb67f40e4033db202326e/metrics.png

Thanks Chris for all this info.  I see you're using NFSv3 so I'll
focus on that, and review all this info for clues.
I also have been working on some updated test cases and see some very
unusual behavior like you're reporting.

I also confirmed that adding the two patches for "Issue #1"  onto
6.4-rc1 resolve _most_ of the caching issues.
However, even after those patches, in some limited instances, there
are still NFS reads over the wire when there should only be reads from
the cache.
There may be multiple bugs here.

I actually misspoke regarding "multiple bugs", as I forgot to add a
small NFS hunk (see below) needed to dhowells 2nd patch (v6 of mm,
netfs, fscache: Stop read optimisation when folio removed from
pagecache).
After the below small hunk was added on top of dhowells 2nd patch, all
my tests pass.
I've also enhanced my existing tests to check NFS READs, fscache
READs, and fscache WRITEs as expected.
And I added an additional test to create files the size of RAM, read
them multiple times, and check for the ops are as expected.
So I'm confident if we add dhowells 2 patches, plus the below hunk for
NFS, these caching issues will be resolved.

diff --git a/fs/nfs/fscache.c b/fs/nfs/fscache.c
index 8c35d88a84b1..d4a20748b14f 100644
--- a/fs/nfs/fscache.c
+++ b/fs/nfs/fscache.c

@@ -180,6 +180,10 @@ void nfs_fscache_init_inode(struct inode *inode)
                                               &auxdata,      /* aux_data */
                                               sizeof(auxdata),
                                               i_size_read(inode));
+
+       if (netfs_inode(inode)->cache)
+               mapping_set_release_always(inode->i_mapping);
+
 }

 /*