On 2021/9/10 下午1:39, Sam Edwards wrote:

Hello again,

I've checked the hardware. The RAM tests out okay and SSD reads seem
consistent. I'd be very surprised if this was hardware failure,
though: the system isn't nearly old enough for degradation and I
haven't seen any of the tell-tale signs of defective hardware.

Unless there is obvious corruption shown from btrfs tree-checker, we
won't even bother considering RAM bitflip.

But for SSD/HDD, we're more cautious about any transid error.

There are already known bad disks which don't follow FLUSH requests,
thus leads to super easy transid mismatch on powerloss:

https://btrfs.wiki.kernel.org/index.php/Hardware_bugs


But in your case, dm-crypto is another layer between btrfs and hardware,
thus we can't verify if it's dm-crypto or btrfs doing something wrong.

Also, if this was due to bit flips in the SSD, the dm-crypt layer
would amplify that across the 128-bit AES blocks (cipher mode is XTS).
I'm not seeing signs of that either.

I'm now pretty familiar with btrfs inspect-internal and have checked
the trees manually. Absolutely nothing looks corrupt, but several
leaf/node blocks are outdated. More interesting is the byte range of
blocks that were "rolled back": 1065332064256-1065565601792, by my
count. That fits pretty nicely in 256 MiB. This range is also not even
close to aligned on a 256 MiB boundary in LBA terms - even taking into
account luks and partition offsets. Thus this seems more like a
software cache that didn't get flushed rather than a SSD bug.

So, question: does the btrfs module use 256 MiB as a default size for
write-back cache pages anywhere?

No.

Or might this problem reside deeper
in the block storage system?

So if possible, please test without dm-crypto to make sure it's really
btrfs causing the problem.

Also, for what it's worth: the last generation to be written before
these inconsistencies seems to be 66303. Files written as part of that
transaction have a birth date of Sep. 7. That's during the same boot
as the failure to read-only,

That's interesting.

Btrfs aborts transaction when it hits critical metadata problems to
prevent further damage, but I still remember some possible corruption
caused by aborting transaction in the old days.
But those corruptions should not exist in v5.14.

Furthermore, the read-only flips itself is more important, do you have
the dmesg of that incident?


Recently we also see some corrupted free space cache causing problems.

If you're going to re-test this, mind to use space cache v2 to do the
tests?

which suggests that the cached buffer
wasn't merely "not saved before a previous shutdown" but rather
"discarded without being written back." Thoughts?

This should not be a thing for btrfs.

Btrfs COW protects its metadata, as long as there is no problem in btrfs
itself allocating new tree blocks to ex you're reporting some RO
incidents, then I guess the COW mechanism may be broken at that time
alreadyisting blocks in the same transaction, we're completely safe.

But considering you're reporting some RO incidents, then I guess the COW
mechanism may be broken at that time already.
(Maybe abort leads to some corrupted free space cache?)

Thanks,
Qu

Cheers,
Sam


On Wed, Sep 8, 2021 at 6:47 PM Sam Edwards [off-list ref] wrote:

quoted

Hello list,

First, I should say that there's no urgency here on my part.
Everything important is very well backed up, and even the
"unimportant" files (various configs) seem readable. I imaged the
partition without even attempting a repair. Normally, my inclination
would be to shrug this off and recreate the filesystem.

However, I'd like to help investigate the root cause, because:
1. This happened suspiciously soon (see my timeline in the link below)
after upgrading to kernel 5.14.1, so may be a serious regression.
2. The filesystem was created less than 5 weeks ago, so the possible
causes are relatively few.
3. My last successful btrfs scrub was just before upgrading to 5.14.1,
hopefully narrowing possible root causes even more.
4. I have imaged the partition and am thus willing to attempt risky
experimental repairs. (Mostly for the sake of reporting if they work.)

Disk setup: NVMe SSD, GPT partition, dm-crypt, btrfs as root fs (no LVM)
OS: Gentoo
Earliest kernel ever used: 5.10.52-gentoo
First kernel version used for "real" usage: 5.13.8
Relevant information: See my Gist,
https://gist.github.com/CFSworks/650280371fc266b2712d02aa2f4c24e8
Misc. notes: I have run "fstrim /" on occasion, but don't have
discards enabled automatically. I doubt TRIM is the culprit, but I
can't rule it out.

My primary hypothesis is that there's some write bug in Linux 5.14.1.
I installed some package updates right before btrfs detected the
problem, and most of the files in the `btrfs check` output look like
they were created as part of those updates.

My secondary hypothesis is that creating and/or using the swapfile
caused some kind of silent corruption that didn't become a detectable
issue until several further writes later.

Let me know if there's anything else I should try/provide!

Regards,
Sam