Thread (35 messages) 35 messages, 13 authors, 2021-01-14

Re: Aarch64 EXT4FS inode checksum failures - seems to be weak memory ordering issues

From: Will Deacon <will@kernel.org>
Date: 2021-01-06 17:21:20
Also in: linux-arm-kernel, lkml

On Wed, Jan 06, 2021 at 01:52:53PM +0000, Russell King - ARM Linux admin wrote:
On Wed, Jan 06, 2021 at 11:53:59AM +0000, Mark Rutland wrote:
quoted
... and are you using defconfig or something else?
Not sure I replied to this. I'm not using the defconfig, I've my own
.config

As I mentioned, Will has built a 5.10 kernel using Arnd's gcc 4.9.4
and hasn't been able to reproduce it. He's sent me his kernel, which
I've booted here, and haven't yet been able to provoke it.

Meanwhile, my 5.9 kernel continues to exhibit this problem, so I've
sent Will my .config (which I'll include here.) There are differences
in some of the block layer configuration. There's differences in the
errata configuration, but we don't think that's a cause (they're not
relevant for Cortex A72).

Our plan is:
- Will is switching to 5.9, and using my config as a base for his
  platform.
- Will is going to send me his modified version of my config.
- We are both going to build using the same kernel sources and same
  config.
- We are going to test our own kernels, and also swap kernel images
  and test each others.

Watch this space for more news...
I've managed to reproduce the corruption on my AMD Seattle board (8x A57).
I haven't had a chance to dig deeper yet, but here's the recipe which works
for me:

1. I'm using GCC 4.9.4 simply to try to get as close as I can to rmk's
   setup. I don't know if this is necessary or not, but the toolchain is
   here:

   https://kernel.org/pub/tools/crosstool/files/bin/arm64/4.9.4/arm64-gcc-4.9.4-nolibc-aarch64-linux-gnu.tar.xz

   and I needed to pull down an old libmpfr to get cc1 to work:

   http://ports.ubuntu.com/pool/main/m/mpfr4/libmpfr4_3.1.2-1_arm64.deb

2. I build a 5.9 kernel with the config here:

   https://mirrors.edge.kernel.org/pub/linux/kernel/people/will/bugs/rmk/config-5.9.0

   and the resulting Image is here:

   https://mirrors.edge.kernel.org/pub/linux/kernel/people/will/bugs/rmk/Image-5.9.0

3. Using that kernel, I boot into a 64-bit Debian 10 filesystem and open a
   couple of terminals over SSH.

4. In one terminal, I run:

   $ while (true); do find /var /usr /bin /sbin -type f -print0 | xargs -0
     md5sum > /dev/null; echo 2 | sudo tee /proc/sys/vm/drop_caches; done

   (note that sudo will prompt you for a password on the first iteration)

5. In the other terminal, I run:

   $ while (true); do ./hackbench ; sleep 1; done

   where hackbench is built from:

   https://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c

   and compiled according to comment in the source code.

With that, I see the following after ten seconds or so:

  EXT4-fs error (device sda2): ext4_lookup:1707: inode #674497: comm md5sum: iget: checksum invalid

Russell, Mark -- does this recipe explode reliably for you too?

Will
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help