From: Russell King - ARM Linux admin <linux@armlinux.org.uk> Date: 2021-01-05 15:48:45
Hi,
This is an update on where I am with this long standing issue at the
current time.
Since 5.4, I have been struggling with several of my ARM64 systems, of
different SoC vendors and differing filesystem media, were sporadically
reporting inode checksum failures on their root filesystems. The time
taken to report this has been anything between a few hours and three
months of uptime, making the problem unrealistic to bisect.
The issue was first seen on my SolidRun Clearfog CX LX2160A based
system, but was also subsequently noticed on my Armada 8040 based
systems running kernels 5.4 and later. Kernel 5.2 has proven stable
with 566 days of uptime with no issue.
It has taken a long time to get debugging in place to see what is going
on - and this is currently detailed on the front page of
www.armlinux.org.uk right now, which has formed a blog of this problem
- since almost no one has taken any interest in it.
However, over the last couple of days, a way to reproduce it has been
found, at least for the LX2160A based system. Power down, leave the
machine powered off for some time. Power up, log in and run:
while :; do sleep 5; find /var /usr /bin /sbin -type f -print0 | \
xargs -0 md5sum >/dev/null; done
Within a few minutes it seems to have spat out an inode checksum
failure if the problem exists. However, testing for the problem _not_
existing is quite difficult - just because it doesn't appear in the
first few minutes does not mean it has been solved - see above where it
can take three months.
However, evidence is currently pointing towards commit 22ec71615d82
("arm64: io: Relax implicit barriers in default I/O accessors") having
revealed this problem. Will is very certain that this change is
correct, and we feel that it may have exposed some other issue in the
Aarch64 code.
Further attempts seem to suggest that the problem is specifically the
barrier in __iormb(). Leaving __iowmb() untouched, and changing the
barrier in __iormb() from dma_rmb() to rmb() _appears_ to result in the
problem disappearing. "Appears" is stressed because further testing is
needed - and that is probably going to take many months before we know
for certain.
However, this suggests that there is a memory ordering bug with aarch64
somewhere. Will can follow up with his own thoughts to this email.
We don't know if it is:
- the kernel.
- the Cortex A72.
- the Cache coherent interconnect.
I don't think it's the CCI, as I believe the Armada 8040 uses Marvell's
own IP for that based around Aurora 2 (the functional spec doesn't make
it clear.) Remember, I'm seeing this problem on both Armada 8040 and
LX2160A. We don't know of any known errata for the A72 in this area.
So, we're down to something in the kernel.
It is possible that it could be compiler related, but I don't see that;
if the "dmb oshld" were strong enough, then it should mean that the
subsequent reads to checksum the inode data after the inode data has
been DMA'd into memory should be reading the correct values from memory
already - but they aren't. And if changing "dmb oshld" to "dsb ld" means
that the code can then read the right values, that to me points fairly
definitively to a hardware problem.
Now, ext4fs is pretty good at checksumming the metadata in the
filesystem - each inode is individually checksummed with CRC32C and two
16-bit halves are stored in each inode. Directories are also
checksummed. ext4fs validates the inode checksum on every ext4_iget()
call. Do other filesystems do similar?
Anyway, here is the patch I'm currently running, which _seems_ so far
to be the minimal fix for my problems. Will thinks that this is hiding
the real problem by adding barriers, but I don't see there's much
choice but to apply this - I don't see what other debugging could be
done without the use of expensive hardware simulation, or detailed
hardware level tracing - the kind of which a silicon vendor or ARM Ltd
would have.
I'm at the end of what I can do with this; I'm going to keep this patch
in my kernel, since it fixes it for me.
Will would like a reliable reproducer - yes, that would be ideal, but
I'm afraid that's a mamoth task in itself. It's taken a year to find
this method of reproducing it.
There's also the matter that in one case I've seen, the ext4 checksum
has been wrong. The subsequent hexdump has been correct, and the post-
hexdump checksum recalculation has remained incorrect - and the same
value as the first incorrect checksum. However, the inode with the
_same_ checksum has subsequently validated correctly by the kernel and
by e2fsck. I can not explain this.
From: Darrick J. Wong <hidden> Date: 2021-01-05 18:30:44
On Tue, Jan 05, 2021 at 03:47:26PM +0000, Russell King - ARM Linux admin wrote:
Hi,
This is an update on where I am with this long standing issue at the
current time.
Since 5.4, I have been struggling with several of my ARM64 systems, of
different SoC vendors and differing filesystem media, were sporadically
reporting inode checksum failures on their root filesystems. The time
taken to report this has been anything between a few hours and three
months of uptime, making the problem unrealistic to bisect.
Aha, I was wondering what happened to this bug report. :)
The issue was first seen on my SolidRun Clearfog CX LX2160A based
system, but was also subsequently noticed on my Armada 8040 based
systems running kernels 5.4 and later. Kernel 5.2 has proven stable
with 566 days of uptime with no issue.
It has taken a long time to get debugging in place to see what is going
on - and this is currently detailed on the front page of
www.armlinux.org.uk right now, which has formed a blog of this problem
- since almost no one has taken any interest in it.
However, over the last couple of days, a way to reproduce it has been
found, at least for the LX2160A based system. Power down, leave the
machine powered off for some time. Power up, log in and run:
while :; do sleep 5; find /var /usr /bin /sbin -type f -print0 | \
xargs -0 md5sum >/dev/null; done
Does that fill up the page cache enough to push memory reclaim?
Within a few minutes it seems to have spat out an inode checksum
failure if the problem exists. However, testing for the problem _not_
existing is quite difficult - just because it doesn't appear in the
first few minutes does not mean it has been solved - see above where it
can take three months.
However, evidence is currently pointing towards commit 22ec71615d82
("arm64: io: Relax implicit barriers in default I/O accessors") having
revealed this problem. Will is very certain that this change is
correct, and we feel that it may have exposed some other issue in the
Aarch64 code.
Further attempts seem to suggest that the problem is specifically the
barrier in __iormb(). Leaving __iowmb() untouched, and changing the
barrier in __iormb() from dma_rmb() to rmb() _appears_ to result in the
problem disappearing. "Appears" is stressed because further testing is
needed - and that is probably going to take many months before we know
for certain.
However, this suggests that there is a memory ordering bug with aarch64
somewhere. Will can follow up with his own thoughts to this email.
We don't know if it is:
- the kernel.
- the Cortex A72.
- the Cache coherent interconnect.
I don't think it's the CCI, as I believe the Armada 8040 uses Marvell's
own IP for that based around Aurora 2 (the functional spec doesn't make
it clear.) Remember, I'm seeing this problem on both Armada 8040 and
LX2160A. We don't know of any known errata for the A72 in this area.
So, we're down to something in the kernel.
It is possible that it could be compiler related, but I don't see that;
if the "dmb oshld" were strong enough, then it should mean that the
subsequent reads to checksum the inode data after the inode data has
been DMA'd into memory should be reading the correct values from memory
already - but they aren't. And if changing "dmb oshld" to "dsb ld" means
that the code can then read the right values, that to me points fairly
definitively to a hardware problem.
Now, ext4fs is pretty good at checksumming the metadata in the
filesystem - each inode is individually checksummed with CRC32C and two
16-bit halves are stored in each inode. Directories are also
checksummed. ext4fs validates the inode checksum on every ext4_iget()
call. Do other filesystems do similar?
XFS and ext4 both validate the ondisk csum when constructing their
incore inodes, and set them when flushing the incore inode back to disk.
I vaguely wonder if there's something else going on here, like ... a
background memory reclaim thread running on one cpu writes out an inode
core with new checksum (because reading the file bumped the atime), and
then another cpu comes along and has to reconstitute the (just
reclaimed) incore inode, but for whatever reason doesn't get the version
that the other cpu just wrote?
That's like 130% speculative though, and note that I have no idea what
the "outer shareable" domain[1] is.
[1] https://developer.arm.com/docs/ddi0597/h/base-instructions-alphabetic-order/dmb-data-memory-barrier
Anyway, here is the patch I'm currently running, which _seems_ so far
to be the minimal fix for my problems. Will thinks that this is hiding
the real problem by adding barriers, but I don't see there's much
choice but to apply this - I don't see what other debugging could be
done without the use of expensive hardware simulation, or detailed
hardware level tracing - the kind of which a silicon vendor or ARM Ltd
would have.
(FWIW I haven't seen checksum errors on xfs or ext4 on arm64, though
most of my testing is relegated to beating on a raspberry pi very
slowly...)
I'm at the end of what I can do with this; I'm going to keep this patch
in my kernel, since it fixes it for me.
Well if you've managed to hit this on multiple different machines after
a long soak time, I wonder how many other people will trip over this too.
It wouldn't be the first time a fs stunts performance to avoid
corruption. ;)
Will would like a reliable reproducer - yes, that would be ideal, but
I'm afraid that's a mamoth task in itself. It's taken a year to find
this method of reproducing it.
There's also the matter that in one case I've seen, the ext4 checksum
has been wrong. The subsequent hexdump has been correct, and the post-
hexdump checksum recalculation has remained incorrect - and the same
value as the first incorrect checksum. However, the inode with the
_same_ checksum has subsequently validated correctly by the kernel and
by e2fsck. I can not explain this.
Strange. You're just using the same ext4_inode_csum() that everything
else uses, right?
--D
From: Russell King - ARM Linux admin <linux@armlinux.org.uk> Date: 2021-01-05 19:50:53
On Tue, Jan 05, 2021 at 10:27:28AM -0800, Darrick J. Wong wrote:
On Tue, Jan 05, 2021 at 03:47:26PM +0000, Russell King - ARM Linux admin wrote:
quoted
Hi,
This is an update on where I am with this long standing issue at the
current time.
Since 5.4, I have been struggling with several of my ARM64 systems, of
different SoC vendors and differing filesystem media, were sporadically
reporting inode checksum failures on their root filesystems. The time
taken to report this has been anything between a few hours and three
months of uptime, making the problem unrealistic to bisect.
Aha, I was wondering what happened to this bug report. :)
Yes... it's continued causing me grief!
quoted
The issue was first seen on my SolidRun Clearfog CX LX2160A based
system, but was also subsequently noticed on my Armada 8040 based
systems running kernels 5.4 and later. Kernel 5.2 has proven stable
with 566 days of uptime with no issue.
It has taken a long time to get debugging in place to see what is going
on - and this is currently detailed on the front page of
www.armlinux.org.uk right now, which has formed a blog of this problem
- since almost no one has taken any interest in it.
However, over the last couple of days, a way to reproduce it has been
found, at least for the LX2160A based system. Power down, leave the
machine powered off for some time. Power up, log in and run:
while :; do sleep 5; find /var /usr /bin /sbin -type f -print0 | \
xargs -0 md5sum >/dev/null; done
Does that fill up the page cache enough to push memory reclaim?
No - I have around 30GB of memory available on the LX2160A system, and
around 3GB that the above is crunching through.
quoted
Within a few minutes it seems to have spat out an inode checksum
failure if the problem exists. However, testing for the problem _not_
existing is quite difficult - just because it doesn't appear in the
first few minutes does not mean it has been solved - see above where it
can take three months.
However, evidence is currently pointing towards commit 22ec71615d82
("arm64: io: Relax implicit barriers in default I/O accessors") having
revealed this problem. Will is very certain that this change is
correct, and we feel that it may have exposed some other issue in the
Aarch64 code.
Further attempts seem to suggest that the problem is specifically the
barrier in __iormb(). Leaving __iowmb() untouched, and changing the
barrier in __iormb() from dma_rmb() to rmb() _appears_ to result in the
problem disappearing. "Appears" is stressed because further testing is
needed - and that is probably going to take many months before we know
for certain.
However, this suggests that there is a memory ordering bug with aarch64
somewhere. Will can follow up with his own thoughts to this email.
We don't know if it is:
- the kernel.
- the Cortex A72.
- the Cache coherent interconnect.
I don't think it's the CCI, as I believe the Armada 8040 uses Marvell's
own IP for that based around Aurora 2 (the functional spec doesn't make
it clear.) Remember, I'm seeing this problem on both Armada 8040 and
LX2160A. We don't know of any known errata for the A72 in this area.
So, we're down to something in the kernel.
It is possible that it could be compiler related, but I don't see that;
if the "dmb oshld" were strong enough, then it should mean that the
subsequent reads to checksum the inode data after the inode data has
been DMA'd into memory should be reading the correct values from memory
already - but they aren't. And if changing "dmb oshld" to "dsb ld" means
that the code can then read the right values, that to me points fairly
definitively to a hardware problem.
Now, ext4fs is pretty good at checksumming the metadata in the
filesystem - each inode is individually checksummed with CRC32C and two
16-bit halves are stored in each inode. Directories are also
checksummed. ext4fs validates the inode checksum on every ext4_iget()
call. Do other filesystems do similar?
XFS and ext4 both validate the ondisk csum when constructing their
incore inodes, and set them when flushing the incore inode back to disk.
I vaguely wonder if there's something else going on here, like ... a
background memory reclaim thread running on one cpu writes out an inode
core with new checksum (because reading the file bumped the atime), and
then another cpu comes along and has to reconstitute the (just
reclaimed) incore inode, but for whatever reason doesn't get the version
that the other cpu just wrote?
That's like 130% speculative though, and note that I have no idea what
the "outer shareable" domain[1] is.
[1] https://developer.arm.com/docs/ddi0597/h/base-instructions-alphabetic-order/dmb-data-memory-barrier
In some of these cases I have observed, the inode appears not to have
changed for quite a long time - it seems to be "media read due to
iget -> checksum failure", although I can't say that definitively.
Given that most of the instances I've been able to provoke over the last
two days have been on the LX2160A, and have been a case of "power on
from cold, boot, login, run the above command, it fails on the first
iteration", it's unlikely that there's much in the way of inode
modification going on.
I've had a few instances now where, for example:
provided = 4b98d7c4 calculated = 5cec3c81
inode(ffffffa6d4dd9c00)
00000000: a4 81 00 00 41 0a 00 00 03 57 f3 5f 25 18 fd 5d
00000010: 70 57 fe 5a 00 00 00 00 00 00 01 00 08 00 00 00
00000020: 00 00 08 00 01 00 00 00 0a f3 01 00 04 00 00 00
00000030: 00 00 00 00 00 00 00 00 01 00 00 00 36 14 11 00
00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000060: 00 00 00 00 1a f4 40 7c 00 00 00 00 00 00 00 00
00000070: 00 00 00 00 00 00 00 00 00 00 00 00 c4 d7 00 00
00000080: 20 00 98 4b a8 c9 54 61 00 00 00 00 40 58 34 9f
00000090: 25 18 fd 5d a8 c9 54 61 00 00 00 00 00 00 00 00
000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
recalculated = 5cec3c81
EXT4-fs error (device nvme0n1p2): ext4_lookup:1707: inode #270077: comm md5sum: iget: checksum invalid
The reason this is interesting is that the hexdump is _correct_ for
the stored checksum (which is 0x4b98d7c4 - stored at hex offsets +83,
+82, +7d, +7c). However, the initial calculation when validating the
inode was incorrect, and the recalculation after dumping the data was
_also_ incorrect.
As mentioned on my writeup on www.armlinux.org.uk (please read it,
there's lots of information there about the history of this and what
has been tried, including the patch for the above output, and I don't
want to keep having to repeat it in emails as different people ask
the same question!) I've already tried disabling the ARM64 accelerated
CRC32 code, and that has made no difference to this problem.
One thing I have recently been trying is to disable preemption within
ext4_inode_csum_verify() - that _seems_ to make it much harder to
reproduce the problem. I've managed it only once despite trying
multiple times. Unfortunately, the filesystem was not mounted
errors=remount-ro, so I couldn't verify the hexdump.
I'm also trying booting with maxcpus=2, so that the kernel is limited
to a single CPU cluster. So far, I haven't been able to provoke the
problem. However, this means absolutely nothing, as past experience
has shown it may take up to three months to occur.
quoted
Anyway, here is the patch I'm currently running, which _seems_ so far
to be the minimal fix for my problems. Will thinks that this is hiding
the real problem by adding barriers, but I don't see there's much
choice but to apply this - I don't see what other debugging could be
done without the use of expensive hardware simulation, or detailed
hardware level tracing - the kind of which a silicon vendor or ARM Ltd
would have.
(FWIW I haven't seen checksum errors on xfs or ext4 on arm64, though
most of my testing is relegated to beating on a raspberry pi very
slowly...)
There's a thought that this problem could have something to do with
migrating across CPUs in different clusters (hence why trying
maxcpus=2). The LX2160A has 16 CPUs - 8 clusters of 2 CPUs. The Armada
8040 has 2 clusters of 2 CPUs. As I understand it, the bcm2711 has four
CPUs which aren't separated into clusters.
This may explain why, /if/ ext4_inode_csum_verify() migrating between
the clusters is important to trigger this, it hasn't been seen on any
Raspberry Pis (I'm also told that Raspbian defaults to 32-bit kernels
which may be another factor why there haven't been people reporting
this.)
quoted
I'm at the end of what I can do with this; I'm going to keep this patch
in my kernel, since it fixes it for me.
Well if you've managed to hit this on multiple different machines after
a long soak time, I wonder how many other people will trip over this too.
It wouldn't be the first time a fs stunts performance to avoid
corruption. ;)
Well, replacing the __iormb() barrier with a dsb appears to either solve
the problem or make it much less likely to happen - although Will Deacon
feels that this is plastering over the problem, and it will come back to
bite in the future.
However, I have put a kernel on one of the affected systems (which is a
long-running system) that revert's Will's 22ec71615d82 commit I mention
above. This /was/ the situation before 5.4-rc1, and it seems to allow
these platforms to work. If that proves to be stable in six months time,
and there has been no other progress on this bug, then as far as I'm
concerned, that is quite simply the required fix for this.
Quite simply, I've given up with Aarch64 over the last year over this;
it doesn't interest me at the moment - not even for development of
other stuff - as fundamentally I have had my trust in it totally
shattered. ARM32 is clearly the superior architecture! :)
quoted
Will would like a reliable reproducer - yes, that would be ideal, but
I'm afraid that's a mamoth task in itself. It's taken a year to find
this method of reproducing it.
There's also the matter that in one case I've seen, the ext4 checksum
has been wrong. The subsequent hexdump has been correct, and the post-
hexdump checksum recalculation has remained incorrect - and the same
value as the first incorrect checksum. However, the inode with the
_same_ checksum has subsequently validated correctly by the kernel and
by e2fsck. I can not explain this.
Strange. You're just using the same ext4_inode_csum() that everything
else uses, right?
From: Mark Rutland <mark.rutland@arm.com> Date: 2021-01-06 11:54:51
On Tue, Jan 05, 2021 at 03:47:26PM +0000, Russell King - ARM Linux admin wrote:
Hi,
Hi Russell,
This is an update on where I am with this long standing issue at the
current time.
Since 5.4, I have been struggling with several of my ARM64 systems, of
different SoC vendors and differing filesystem media, were sporadically
reporting inode checksum failures on their root filesystems. The time
taken to report this has been anything between a few hours and three
months of uptime, making the problem unrealistic to bisect.
However, over the last couple of days, a way to reproduce it has been
found, at least for the LX2160A based system. Power down, leave the
machine powered off for some time. Power up, log in and run:
while :; do sleep 5; find /var /usr /bin /sbin -type f -print0 | \
xargs -0 md5sum >/dev/null; done
I've just set this off on an Raspberry Pi 4 running a locally-built
arm64 v5.10 defconfig. I'm using a SATA SSD mounted via a USB-SATA
adapter. I'll try to give that a few reboots see if I can reproduce the
issue.
Just to check -- how is your ext4 fs mounted? Mine is mounted as:
/dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro)
... and are you using defconfig or something else?
[...]
It is possible that it could be compiler related, but I don't see that;
if the "dmb oshld" were strong enough, then it should mean that the
subsequent reads to checksum the inode data after the inode data has
been DMA'd into memory should be reading the correct values from memory
already - but they aren't. And if changing "dmb oshld" to "dsb ld" means
that the code can then read the right values, that to me points fairly
definitively to a hardware problem.
Just in case the compiler has some impact, can you way which compilers
you've been using? On the rpi4 I'm using GCC 8.3.0 as packaged by Debian
Buster (10.7).
Thanks,
Mark.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Russell King - ARM Linux admin <linux@armlinux.org.uk> Date: 2021-01-06 12:14:58
On Wed, Jan 06, 2021 at 11:53:59AM +0000, Mark Rutland wrote:
On Tue, Jan 05, 2021 at 03:47:26PM +0000, Russell King - ARM Linux admin wrote:
quoted
Hi,
Hi Russell,
quoted
This is an update on where I am with this long standing issue at the
current time.
Since 5.4, I have been struggling with several of my ARM64 systems, of
different SoC vendors and differing filesystem media, were sporadically
reporting inode checksum failures on their root filesystems. The time
taken to report this has been anything between a few hours and three
months of uptime, making the problem unrealistic to bisect.
quoted
However, over the last couple of days, a way to reproduce it has been
found, at least for the LX2160A based system. Power down, leave the
machine powered off for some time. Power up, log in and run:
while :; do sleep 5; find /var /usr /bin /sbin -type f -print0 | \
xargs -0 md5sum >/dev/null; done
I've just set this off on an Raspberry Pi 4 running a locally-built
arm64 v5.10 defconfig. I'm using a SATA SSD mounted via a USB-SATA
adapter. I'll try to give that a few reboots see if I can reproduce the
issue.
Will, Arnd and myself have been working on this since yesterday - we
_think_ it might have something to do with the clustering of the CPUs.
The LX2160A has 8 clusters of two CPUs. The Armada 8040 has two clusters
of two CPUs. RPi doesn't have clustering.
I've been able to reproduce it on the LX2160A with CPUs 0 and 15, or
CPUs 0 and 4. I haven't been able to reproduce it with CPUs 0 and 1.
However, contary to that, I haven't yet been able to reproduce it with
CPUs 0 and 2.
Just to check -- how is your ext4 fs mounted? Mine is mounted as:
/dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro)
I've tried just plain "rw,errors=remount-ro" and the above.
Another bit of information I have just discovered - it's not Will's
commit. I reverted Will's commit, booted it on the gt8k system
yesterday. This morning, while checking the mount options there, I
noticed the rootfs had gone read only... checking the kernel log...
[45625.530075] provided = 41746cdc calculated = fb8b1ca1
[45625.533858] inode(ffffff82889d5600)
[45625.536090] 00000000: ff a1 00 00 0c 00 00 00 02 ac f3 5f e6 0b 55 5e
[45625.541269] 00000010: 52 cb d1 5c 00 00 00 00 00 00 01 00 00 00 00 00
[45625.546443] 00000020: 00 00 00 00 01 00 00 00 6d 71 5f 73 65 6e 64 2e
[45625.551618] 00000030: 33 2e 67 7a 00 00 00 00 00 00 00 00 00 00 00 00
[45625.556791] 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[45625.561965] 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[45625.567137] 00000060: 00 00 00 00 60 70 be da 00 00 00 00 00 00 00 00
[45625.572316] 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 dc 6c 00 00
[45625.577503] 00000080: 20 00 74 41 34 8e f5 52 00 00 00 00 00 38 9c 1c
[45625.582717] 00000090: e6 0b 55 5e 34 8e f5 52 00 00 00 00 00 00 00 00
[45625.587905] 000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[45625.593083] 000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[45625.598255] 000000c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[45625.603430] 000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[45625.608618] 000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[45625.613795] 000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[45625.618968] recalculated = fb8b1ca1
[45625.621179] EXT4-fs error (device mmcblk0p1): ext4_lookup:1707: inode #149495: comm mandb: iget: checksum invalid
debugfs: stat <149495>
stat: Inode checksum does not match inode while reading inode 149495
Okay, so the data in memory is incorrect...
# dd if=/dev/mmcblk0p1 bs=4096 count=1 skip=525567 | hexdump -C |less
00000600 ff a1 00 00 0c 00 00 00 02 ac f3 5f e6 0b 55 5e |..........._..U^|
00000610 52 cb d1 5c 00 00 00 00 00 00 01 00 00 00 00 00 |R..\............|
00000620 00 00 00 00 01 00 00 00 6d 71 5f 73 65 6e 64 2e |........mq_send.|
00000630 33 2e 67 7a 00 00 00 00 00 00 00 00 00 00 00 00 |3.gz............|
00000640 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000660 00 00 00 00 60 70 be da 00 00 00 00 00 00 00 00 |....`p..........|
00000670 00 00 00 00 00 00 00 00 00 00 00 00 dc 6c 00 00 |.............l..|
00000680 20 00 74 41 34 8e f5 52 00 00 00 00 00 38 9c 1c | .tA4..R.....8..|
00000690 e6 0b 55 5e 34 8e f5 52 00 00 00 00 00 00 00 00 |..U^4..R........|
000006a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
# echo 1 > /proc/sys/vm/drop_caches
# debugfs /dev/mmcblk0p1
debugfs 1.44.5 (15-Dec-2018)
/dev/mmcblk0p1: Inode bitmap checksum does not match bitmap while reading allocation bitmaps
# e2fsck -n /dev/mmcblk0p1
e2fsck 1.44.5 (15-Dec-2018)
Warning: skipping journal recovery because doing a read-only filesystem
check.
/dev/mmcblk0p1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 149495 passes checks, but checksum does not match inode. Fix? no
So in this case, it looks like the inode has been written back to media
with a bad checksum... which is new.
... and are you using defconfig or something else?
Definitely "something else".
quoted
It is possible that it could be compiler related, but I don't see that;
if the "dmb oshld" were strong enough, then it should mean that the
subsequent reads to checksum the inode data after the inode data has
been DMA'd into memory should be reading the correct values from memory
already - but they aren't. And if changing "dmb oshld" to "dsb ld" means
that the code can then read the right values, that to me points fairly
definitively to a hardware problem.
Just in case the compiler has some impact, can you way which compilers
you've been using? On the rpi4 I'm using GCC 8.3.0 as packaged by Debian
Buster (10.7).
I've used 4.9.2 and then switched to 4.9.4. Will has built a kernel
using 4.9.4 which I'm about to try on the lx2160a.
I do _not_ want to try a later compiler and have this problem "vanish".
We will not know whether the problem has been hidden by different code
generation, but still actually exists.
As I've said, my confidence level with Aarch64 as a viable archtecture
is at rock bottom right now, and a definitive explanation needs to be
found for this, otherwise I'm going to dispose of *all* my Aarch64
hardware - I have no confidence anymore that the architecture is
capable of operating safely with modern Linux kernels, and I'd rather
use x86 instead right now. Yes, it this is _that_ serious a problem
for me.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel