Hello,

On Tue, Jul 20, 2021 at 08:56:21AM +0200, Martin Steigerwald wrote:

Qu Wenruo - 18.07.21, 02:16:19 CEST:

quoted

On 2021/7/17 下午4:31, Martin Steigerwald wrote:

quoted

Martin Steigerwald - 16.07.21, 17:19:55 CEST:

quoted

Martin Steigerwald - 16.07.21, 17:05:59 CEST:

quoted

I migrated to a different laptop and this one has a 2TB Samsung 980 Pro drive
(not a 2TB Samsung 870 Evo Plus which previously had problems).

Kernel is:

% cat /proc/version
Linux version 5.13.1-t14 (martin@[…]) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #14 SMP PREEMPT Mon Jul 12 10:36:54 CEST 2021

Another addition that might be important. I am using xxhash.

I created the filesystem like that:

% mkfs.btrfs -L home --csum xxhash /dev/nvme/home

So it is xxhash, discard=async and space cache v2.

My educiated guess is async.

The reason is that all affected tree blocks are showing a fixed pattern:
have 0x48be03222606a29d expected csum 0x0100000026004000

Thank you.

Now it gets interesting. I just found, as I wanted to disable discard,
that it was not enabled to begin with:

% grep home /etc/fstab
/dev/nvme/home  /home           btrfs   lazytime,compress=zstd                                  0       0
[…]

It appears that out of caution after my experiences with Samsung 870 Evo
Plus I went without "discard=async".

Also I stated the exact mount options further down in the mail. No
discard in there:

quoted

quoted

quoted

quoted

It is BTRFS single profile on LVM on LUKS. Mount options are:

rw,relatime,lazytime,compress=zstd:3,ssd,space_cache=v2,subvolid=1054,subvol=/home

So that part of my mail where I claimed I was using "discard=async" was
inaccurate.

quoted

This pretty much means those tree blocks have been discarded.

But I'm not familiar with that part.

Dennis is more familiar with this, maybe he could provide some idea on this.

Any other insights?

Sorry for the late reply.

I'm not particularly involved with btrfs these days. It seems there has
been some changes but nothing major looking at the review log of
fs/btrfs/discard.c.

This may be more insightful for Qu, but here's just a quick idea behind
async discard. The block_groups are what manage allocated space on disk
while the free_space_cache is effectively the negative of that per
block_group. I didn't do a disk modification so the free_space_cache
doesn't track the async state and will bring up a newly mounted file
system as completely untrimmed and async discard will do it's thing.

Async discard traverses the free_space_cache of block_groups and
performs allocation, trim, and freeing. By doing it this way, we avoid
racing with allocations and can live outside the transaction commit
lifecycle.

Sync discard operates on the freed region during transaction commit
before returning it to the free_space_cache.

Given the above, I find it hard that async discard would be directly
responsible for what you're seeing as it just relies on the
free_space_cache accounting. It's possible though there might be
something wrong with the on disk free_space_cache representation and
that would lead to potentially something like this.

Thanks,
Dennis

Best,
Martin

quoted

Thanks,
Qu

quoted

Maybe something in this combination is not yet fully stable?

https://btrfs.wiki.kernel.org/index.php/Status says additional checksum
algorithms are stable. It also states free space cache is stable. And it
states that asynchronous discards are stable. It does not explicitly
mention xxhash or free space v2 are stable too. I bet it may be included
in the general statement, but I am not completely sure.

However what I just did is:

% mount -o remount,clear_cache,space_cache=v2 /home

And I now get:

% btrfs scrub status /home
UUID:             […]
Scrub started:    Sat Jul 17 10:24:54 2021
Status:           finished
Duration:         0:01:43
Total to scrub:   178.39GiB
Rate:             1.73GiB/s
Error summary:    no errors found

Hopefully it will stay this way. Fingers crossed.

So a good trick in case there is no file mentioned in kernal log may be to clear
free space tree and see whether the checksum errors go away.

If anyone can make any additional sense out of this, please go ahead.

quoted

quoted

I thought this time I would be fine, but I just got:

[63168.287911] BTRFS warning (device dm-3): csum failed root 1372 ino 2295743 off 2718461952 csum 0x48be03222606a29d expected csum 0x0100000026004000 mirror 1
[63168.287925] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[63168.346552] BTRFS warning (device dm-3): csum failed root 1372 ino 2295743 off 2718461952 csum 0x48be03222606a29d expected csum 0x0100000026004000 mirror 1
[63168.346567] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[63168.346685] BTRFS warning (device dm-3): csum failed root 1372 ino 2295743 off 2718461952 csum 0x48be03222606a29d expected csum 0x0100000026004000 mirror 1
[63168.346708] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[63168.346859] BTRFS warning (device dm-3): csum failed root 1372 ino 2295743 off 2718461952 csum 0x48be03222606a29d expected csum 0x0100000026004000 mirror 1
[63168.346873] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[63299.490367] BTRFS warning (device dm-3): csum failed root 1372 ino 2295743 off 2718461952 csum 0x48be03222606a29d expected csum 0x0100000026004000 mirror 1
[63299.490384] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
[63299.572849] BTRFS warning (device dm-3): csum failed root 1372 ino 2295743 off 2718461952 csum 0x48be03222606a29d expected csum 0x0100000026004000 mirror 1
[63299.572866] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 6, gen 0
[63299.573151] BTRFS warning (device dm-3): csum failed root 1372 ino 2295743 off 2718461952 csum 0x48be03222606a29d expected csum 0x0100000026004000 mirror 1
[63299.573168] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 7, gen 0
[63299.573286] BTRFS warning (device dm-3): csum failed root 1372 ino 2295743 off 2718461952 csum 0x48be03222606a29d expected csum 0x0100000026004000 mirror 1
[63299.573295] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 8, gen 0
[63588.902631] BTRFS warning (device dm-3): csum failed root 1372 ino 4895964 off 34850111488 csum 0x21941ce6e9739bd6 expected csum 0xc113140701000000 mirror 1
[63588.902647] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 13, gen 0
[63588.949614] BTRFS warning (device dm-3): csum failed root 1372 ino 4895964 off 34850111488 csum 0x21941ce6e9739bd6 expected csum 0xc113140701000000 mirror 1
[63588.949628] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 14, gen 0
[63588.949849] BTRFS warning (device dm-3): csum failed root 1372 ino 4895964 off 34850111488 csum 0x21941ce6e9739bd6 expected csum 0xc113140701000000 mirror 1
[63588.949855] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 15, gen 0
[63588.950087] BTRFS warning (device dm-3): csum failed root 1372 ino 4895964 off 34850111488 csum 0x21941ce6e9739bd6 expected csum 0xc113140701000000 mirror 1
[63588.950099] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 16, gen 0

Additional errors revealed through scrubbing – will test the other
filesystems as well:

% btrfs scrub status /home
UUID:             […]
Scrub started:    Fri Jul 16 17:08:49 2021
Status:           finished
Duration:         0:02:05
Total to scrub:   203.54GiB
Rate:             1.63GiB/s
Error summary:    csum=5
   Corrected:      0
   Uncorrectable:  5
   Unverified:     0

Now totalling to 21 errors:

% btrfs device stats /home
[/dev/mapper/nvme-home].write_io_errs    0
[/dev/mapper/nvme-home].read_io_errs     0
[/dev/mapper/nvme-home].flush_io_errs    0
[/dev/mapper/nvme-home].corruption_errs  21
[/dev/mapper/nvme-home].generation_errs  0

Log:

[64707.979036] BTRFS info (device dm-3): scrub: started on devid 1
[64730.009687] BTRFS warning (device dm-3): checksum error at logical 36997591040 on dev /dev/mapper/nvme-home, physical 34850107392, root 1054, inode 2295743, offset 2718461952: path resolving failed with ret=-2
[64730.009710] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 17, gen 0
[64730.009721] BTRFS error (device dm-3): unable to fixup (regular) error at logical 36997591040 on dev /dev/mapper/nvme-home
[64730.010996] BTRFS warning (device dm-3): checksum error at logical 36997595136 on dev /dev/mapper/nvme-home, physical 34850111488, root 1054, inode 4895964, offset 7676579840: path resolving failed with ret=-2
[64730.011014] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 18, gen 0
[64730.011024] BTRFS error (device dm-3): unable to fixup (regular) error at logical 36997595136 on dev /dev/mapper/nvme-home
[64730.011298] BTRFS warning (device dm-3): checksum error at logical 36997599232 on dev /dev/mapper/nvme-home, physical 34850115584, root 1054, inode 4895964, offset 7676579840: path resolving failed with ret=-2
[64730.011312] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 19, gen 0
[64730.011319] BTRFS error (device dm-3): unable to fixup (regular) error at logical 36997599232 on dev /dev/mapper/nvme-home
[64730.011603] BTRFS warning (device dm-3): checksum error at logical 36997603328 on dev /dev/mapper/nvme-home, physical 34850119680, root 1054, inode 4895964, offset 7676579840: path resolving failed with ret=-2
[64730.011616] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 20, gen 0
[64730.011623] BTRFS error (device dm-3): unable to fixup (regular) error at logical 36997603328 on dev /dev/mapper/nvme-home
[64730.011905] BTRFS warning (device dm-3): checksum error at logical 36997607424 on dev /dev/mapper/nvme-home, physical 34850123776, root 1054, inode 4895964, offset 7676579840: path resolving failed with ret=-2
[64730.011921] BTRFS error (device dm-3): bdev /dev/mapper/nvme-home errs: wr 0, rd 0, flush 0, corrupt 21, gen 0
[64730.011928] BTRFS error (device dm-3): unable to fixup (regular) error at logical 36997607424 on dev /dev/mapper/nvme-home
[64832.447560] BTRFS info (device dm-3): scrub: finished on devid 1 with status: 0

Why is BTRFS unable to determine a path?

How would I fix those when BTRFS does not tell me what file is affected?

quoted

during a backup.

According to rsync this is related (why does BTRFS does not report the
affected file?)

Create a snapshot of '/home' in '/zeit/home/backup-2021-07-16-16:40:13'
rsync: [sender] read errors mapping "/zeit/home/backup-2021-07-16-16:40:13/martin/.local/share/akonadi/search_db/email/postlist.glass": Input/output error (5)
rsync: [sender] read errors mapping "/zeit/home/backup-2021-07-16-16:40:13/martin/.local/share/akonadi/search_db/email/postlist.glass": Input/output error (5)
ERROR: martin/.local/share/akonadi/search_db/email/postlist.glass failed verification -- update discarded.
rsync: [sender] read errors mapping "/zeit/home/backup-2021-07-16-16:40:13/martin/.local/share/baloo/index": Input/output error (5)
rsync: [sender] read errors mapping "/zeit/home/backup-2021-07-16-16:40:13/martin/.local/share/baloo/index": Input/output error (5)
ERROR: martin/.local/share/baloo/index failed verification -- update discarded.

Both are frequently written to files (both Baloo and Akonadi have very crazy
I/O patterns that, I would not have thought so, can even satisfy an NVMe SSD).

I thought that a Samsung 980 Pro can easily handle "discard=async" so I
used it.

This is on a ThinkPad T14 Gen1 with AMD Ryzen 7 PRO 4750U and 32 GiB of RAM.

It is BTRFS single profile on LVM on LUKS. Mount options are:

rw,relatime,lazytime,compress=zstd:3,ssd,space_cache=v2,subvolid=1054,subvol=/home

Smartctl has no errors.

I only use a few (less than 10) subvolumes.

I do not have any other errors in kernel log, so I bet this may not be
"discard=async" related. Any idea?

Maybe I still remove "discard=async" for now. Maybe it is not yet fully reliable.

quoted

Could it have to do with a sudden switching off the laptop (there had
been quite some reasons cause at least with a AMD model of this laptop
in combination with an USB-C dock by Lenovo there are quite some stability
issues)? I would have hoped that the Samsung 980 Pro would still be
equipped to complete the outstanding write operation, but maybe it has
no capacitor for this.

I am really surprised by the what I experienced about the reliability of
SSDs I recently bought. I did not see a failure within a month with any
of the older SSDs. I hope this does not point at a severe worsening of
the quality. Probably I have to fit another SSD in there and use BTRFS
RAID 1 again to protect at least part of the data from errors like this.

Any idea about this? I bet you may not have any, as there is not block
I/O related errors in the log, but if you have, by all means share your
thoughts. Thank you.

Both files can be recreated. So I bet I will just remove them.

Best,

-- 
Martin