[bisected] Re: odd NFS v4.2 regression 4.4..5.15

On Tue, 2021-10-12 at 15:54 +0200, Mike Galbraith wrote:

Greetings network wizards,

I seem to have stumbled across a kind of odd looking NFS regression
while looking at something unrelated.

I have a few VMs that are mirrors of my desktop box, including some
data, but they usually over-mount directories with tons of data via
NFS, thus hang off the host like a mini-me Siamese twin for testing.

In the data below, I run a script in git-daemon's home to extract RT
patch sets with git-format-patch, on the host for reference, on the
host in an NFS mount of itself, and the mini-me VM, where I started.

Note: bonnie throughput looked fine for v4.2.

host
time su - git -c ./format-rt-patches.sh
real    1m5.923s
user    0m54.692s
sys     0m11.550s

NFS mount my own box, and do the same from the same spot
time su git -c 'cd /homer/home/git;./format-rt-patches.sh'

4.4.231     v4.2        v3
1   real    2m27.046s   2m2.059s
    user    0m59.190s   0m58.701s
    sys     0m41.448s   0m32.541s

5.15-rc5    v4.2        v3
   real     3m14.954s   2m8.366s
   user     0m59.901s   0m58.317s
   sys      0m52.708s   0m32.313s
        vs 1   0.754       0.951

repeats     v4.2        v3
   real     3m16.313s   2m7.940s
   real     3m10.905s   2m8.029s

Found the v4.2 regression (for this load anyway) to have landed in 5.4.
It then bisected cleanly to either the build bug or its neighbor.

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189
eb82dd393744107ebc365a53e7813c7c67cb203b
We cannot bisect more!

git bisect start
# good: [4d856f72c10ecb060868ed10ff1b1453943fc6c8] Linux 5.3
git bisect good 4d856f72c10ecb060868ed10ff1b1453943fc6c8
# bad: [219d54332a09e8d8741c1e1982f5eae56099de85] Linux 5.4
git bisect bad 219d54332a09e8d8741c1e1982f5eae56099de85
# good: [a9f8b38a071b468276a243ea3ea5a0636e848cf2] Merge tag 'for-linus-5.4-1' of git://github.com/cminyard/linux-ipmi
git bisect good a9f8b38a071b468276a243ea3ea5a0636e848cf2
# good: [5825a95fe92566ada2292a65de030850b5cff1da] Merge tag 'selinux-pr-20190917' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/selinux
git bisect good 5825a95fe92566ada2292a65de030850b5cff1da
# bad: [48acba989ed5d8707500193048d6c4c5945d5f43] Merge tag 'riscv/for-v5.4-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux
git bisect bad 48acba989ed5d8707500193048d6c4c5945d5f43
# good: [7bccb9f10c8f36ee791769b531ed4d28f6379aae] Merge tag 'linux-watchdog-5.4-rc1' of git://www.linux-watchdog.org/linux-watchdog
git bisect good 7bccb9f10c8f36ee791769b531ed4d28f6379aae
# bad: [ef129d34149ea23d0d442844fc25ae26a85589fc] selftests/net: add nettest to .gitignore
git bisect bad ef129d34149ea23d0d442844fc25ae26a85589fc
# bad: [9c5efe9ae7df78600c0ee7bcce27516eb687fa6e] Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 9c5efe9ae7df78600c0ee7bcce27516eb687fa6e
# good: [8bbe0dec38e147a50e9dd5f585295f7e68e0f2d0] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
git bisect good 8bbe0dec38e147a50e9dd5f585295f7e68e0f2d0
# bad: [298fb76a5583900a155d387efaf37a8b39e5dea2] Merge tag 'nfsd-5.4' of git://linux-nfs.org/~bfields/linux
git bisect bad 298fb76a5583900a155d387efaf37a8b39e5dea2
# bad: [bbf2f098838aa86cee240a7a11486e0601d56a3f] nfsd: Reset the boot verifier on all write I/O errors
git bisect bad bbf2f098838aa86cee240a7a11486e0601d56a3f
# skip: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
git bisect skip fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189
# bad: [5e113224c17e2fb156b785ddbbc48a0209fddb0c] nfsd: nfsd_file cache entries should be per net namespace
git bisect bad 5e113224c17e2fb156b785ddbbc48a0209fddb0c
# good: [48cd7b51258c1a158293cefb97dda988080f5e13] nfsd: hook up nfsd_read to the nfsd_file cache
git bisect good 48cd7b51258c1a158293cefb97dda988080f5e13
# bad: [501cb1849f865960501d19d54e6a5af306f9b6fd] nfsd: rip out the raparms cache
git bisect bad 501cb1849f865960501d19d54e6a5af306f9b6fd
# bad: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
git bisect bad eb82dd393744107ebc365a53e7813c7c67cb203b
# good: [5920afa3c85ff38642f652b6e3880e79392fcc89] nfsd: hook nfsd_commit up to the nfsd_file cache
git bisect good 5920afa3c85ff38642f652b6e3880e79392fcc89
# only skipped commits left to test
# possible first bad commit: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
# possible first bad commit: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
# good: [5920afa3c85ff38642f652b6e3880e79392fcc89] nfsd: hook nfsd_commit up to the nfsd_file cache
git bisect good 5920afa3c85ff38642f652b6e3880e79392fcc89
# only skipped commits left to test
# possible first bad commit: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
# possible first bad commit: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files