Re: Aarch64 EXT4FS inode checksum failures - seems to be weak memory ordering issues
From: Will Deacon <will@kernel.org>
Date: 2021-01-06 17:21:20
Also in:
linux-arm-kernel, lkml
On Wed, Jan 06, 2021 at 01:52:53PM +0000, Russell King - ARM Linux admin wrote:
On Wed, Jan 06, 2021 at 11:53:59AM +0000, Mark Rutland wrote:quoted
... and are you using defconfig or something else?Not sure I replied to this. I'm not using the defconfig, I've my own .config As I mentioned, Will has built a 5.10 kernel using Arnd's gcc 4.9.4 and hasn't been able to reproduce it. He's sent me his kernel, which I've booted here, and haven't yet been able to provoke it. Meanwhile, my 5.9 kernel continues to exhibit this problem, so I've sent Will my .config (which I'll include here.) There are differences in some of the block layer configuration. There's differences in the errata configuration, but we don't think that's a cause (they're not relevant for Cortex A72). Our plan is: - Will is switching to 5.9, and using my config as a base for his platform. - Will is going to send me his modified version of my config. - We are both going to build using the same kernel sources and same config. - We are going to test our own kernels, and also swap kernel images and test each others. Watch this space for more news...
I've managed to reproduce the corruption on my AMD Seattle board (8x A57). I haven't had a chance to dig deeper yet, but here's the recipe which works for me: 1. I'm using GCC 4.9.4 simply to try to get as close as I can to rmk's setup. I don't know if this is necessary or not, but the toolchain is here: https://kernel.org/pub/tools/crosstool/files/bin/arm64/4.9.4/arm64-gcc-4.9.4-nolibc-aarch64-linux-gnu.tar.xz and I needed to pull down an old libmpfr to get cc1 to work: http://ports.ubuntu.com/pool/main/m/mpfr4/libmpfr4_3.1.2-1_arm64.deb 2. I build a 5.9 kernel with the config here: https://mirrors.edge.kernel.org/pub/linux/kernel/people/will/bugs/rmk/config-5.9.0 and the resulting Image is here: https://mirrors.edge.kernel.org/pub/linux/kernel/people/will/bugs/rmk/Image-5.9.0 3. Using that kernel, I boot into a 64-bit Debian 10 filesystem and open a couple of terminals over SSH. 4. In one terminal, I run: $ while (true); do find /var /usr /bin /sbin -type f -print0 | xargs -0 md5sum > /dev/null; echo 2 | sudo tee /proc/sys/vm/drop_caches; done (note that sudo will prompt you for a password on the first iteration) 5. In the other terminal, I run: $ while (true); do ./hackbench ; sleep 1; done where hackbench is built from: https://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c and compiled according to comment in the source code. With that, I see the following after ten seconds or so: EXT4-fs error (device sda2): ext4_lookup:1707: inode #674497: comm md5sum: iget: checksum invalid Russell, Mark -- does this recipe explode reliably for you too? Will