Re: Corruption suspiciously soon after upgrade to 5.14.1; filesystem less than 5 weeks old

Hello again,

I've checked the hardware. The RAM tests out okay and SSD reads seem
consistent. I'd be very surprised if this was hardware failure,
though: the system isn't nearly old enough for degradation and I
haven't seen any of the tell-tale signs of defective hardware.

Also, if this was due to bit flips in the SSD, the dm-crypt layer
would amplify that across the 128-bit AES blocks (cipher mode is XTS).
I'm not seeing signs of that either.

I'm now pretty familiar with btrfs inspect-internal and have checked
the trees manually. Absolutely nothing looks corrupt, but several
leaf/node blocks are outdated. More interesting is the byte range of
blocks that were "rolled back": 1065332064256-1065565601792, by my
count. That fits pretty nicely in 256 MiB. This range is also not even
close to aligned on a 256 MiB boundary in LBA terms - even taking into
account luks and partition offsets. Thus this seems more like a
software cache that didn't get flushed rather than a SSD bug.

So, question: does the btrfs module use 256 MiB as a default size for
write-back cache pages anywhere? Or might this problem reside deeper
in the block storage system?

Also, for what it's worth: the last generation to be written before
these inconsistencies seems to be 66303. Files written as part of that
transaction have a birth date of Sep. 7. That's during the same boot
as the failure to read-only, which suggests that the cached buffer
wasn't merely "not saved before a previous shutdown" but rather
"discarded without being written back." Thoughts?

Cheers,
Sam


On Wed, Sep 8, 2021 at 6:47 PM Sam Edwards [off-list ref] wrote:

Hello list,

First, I should say that there's no urgency here on my part.
Everything important is very well backed up, and even the
"unimportant" files (various configs) seem readable. I imaged the
partition without even attempting a repair. Normally, my inclination
would be to shrug this off and recreate the filesystem.

However, I'd like to help investigate the root cause, because:
1. This happened suspiciously soon (see my timeline in the link below)
after upgrading to kernel 5.14.1, so may be a serious regression.
2. The filesystem was created less than 5 weeks ago, so the possible
causes are relatively few.
3. My last successful btrfs scrub was just before upgrading to 5.14.1,
hopefully narrowing possible root causes even more.
4. I have imaged the partition and am thus willing to attempt risky
experimental repairs. (Mostly for the sake of reporting if they work.)

Disk setup: NVMe SSD, GPT partition, dm-crypt, btrfs as root fs (no LVM)
OS: Gentoo
Earliest kernel ever used: 5.10.52-gentoo
First kernel version used for "real" usage: 5.13.8
Relevant information: See my Gist,
https://gist.github.com/CFSworks/650280371fc266b2712d02aa2f4c24e8
Misc. notes: I have run "fstrim /" on occasion, but don't have
discards enabled automatically. I doubt TRIM is the culprit, but I
can't rule it out.

My primary hypothesis is that there's some write bug in Linux 5.14.1.
I installed some package updates right before btrfs detected the
problem, and most of the files in the `btrfs check` output look like
they were created as part of those updates.

My secondary hypothesis is that creating and/or using the swapfile
caused some kind of silent corruption that didn't become a detectable
issue until several further writes later.

Let me know if there's anything else I should try/provide!

Regards,
Sam