On 21/05/10 04:38PM, Qu Wenruo wrote:

Hi Ritesh,

I guess no error report so far is a good thing?

Sorry about the delay in starting of my testing. Was not keeping well since
Friday onwards, hence could not start the testing. (Feeling much better now).

So -g quick passed w/o any fatal issues. But with -g auto I got a kernel bug
with btrfs/28. Below is the report.

Just to report what my result is, I ran my latest github branch for the
full weekend, over 50 hours, and around 20 runs of full generic/auto
without defrag groups.

And I see no crash at all.

But there is a special note, there is a new patch, introduced just
before the weekend (Fri May 7 09:31:43 2021 +0800), titled "btrfs: fix a
possible use-after-free race in metadata read path", is a new fix for a
bug I reproduced once locally.

Yes,  I already have this in my tree. This is the latest patch in my tree which
I am testing.
"btrfs: remove io_failure_record::in_validation"

The bug should only happen when read is slow and only happens for
metadata read path.

The details can be found in the commit message, although it's rare to
hit, I have hit such problem around 3 times in total.

Hopes you didn't hit any crash during your test.

I am hitting below bug_on(). Since I saw your email just now, so I am directly
reporting this failure, w/o analyzing. Please let me know if you need anything
else from my end for this.

I will halt the testing of "-g auto" for now. Once we have some conclusion on
this one, then will resume the testing.

btrfs/028 32s ... 	[10:41:18][  780.104573] run fstests btrfs/028 at 2021-05-10 10:41:18

[  780.732073] BTRFS: device fsid be9b827d-28ee-4a5e-80a0-e19971061a58 devid 1 transid 5 /dev/vdc scanned by mkfs.btrfs (21129)
[  780.759754] BTRFS info (device vdc): disk space caching is enabled
[  780.759848] BTRFS info (device vdc): has skinny extents
[  780.759888] BTRFS warning (device vdc): read-write for sector size 4096 with page size 65536 is experimental
<...>
[  784.580404] BTRFS info (device vdc): found 21 extents, stage: move data extents
[  784.878376] BTRFS info (device vdc): found 13 extents, stage: update data pointers
[  785.175349] BTRFS info (device vdc): balance: ended with status: 0
[  785.367729] BTRFS info (device vdc): balance: start -d
[  785.400884] BTRFS info (device vdc): relocating block group 2446327808 flags data
[  785.527858] btrfs_print_data_csum_error: 18 callbacks suppressed
[  785.527865] BTRFS warning (device vdc): csum failed root -9 ino 262 off 393216 csum 0x8941f998 expected csum 0x9439dda4 mirror 1
[  785.528406] btrfs_dev_stat_print_on_error: 18 callbacks suppressed
[  785.528409] BTRFS error (device vdc): bdev /dev/vdc errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[  785.528857] BTRFS warning (device vdc): csum failed root -9 ino 262 off 397312 csum 0x8941f998 expected csum 0x9439dda4 mirror 1
[  785.529166] BTRFS error (device vdc): bdev /dev/vdc errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[  785.529412] BTRFS warning (device vdc): csum failed root -9 ino 262 off 401408 csum 0x8941f998 expected csum 0x667b7e1e mirror 1
[  785.529714] BTRFS error (device vdc): bdev /dev/vdc errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[  785.530321] BTRFS warning (device vdc): csum failed root -9 ino 262 off 393216 csum 0x8941f998 expected csum 0x9439dda4 mirror 1
[  785.530637] BTRFS error (device vdc): bdev /dev/vdc errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[  785.530882] BTRFS warning (device vdc): csum failed root -9 ino 262 off 397312 csum 0x8941f998 expected csum 0x9439dda4 mirror 1
[  785.531185] BTRFS error (device vdc): bdev /dev/vdc errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
[  785.531428] BTRFS warning (device vdc): csum failed root -9 ino 262 off 401408 csum 0x8941f998 expected csum 0x667b7e1e mirror 1
[  785.531719] BTRFS error (device vdc): bdev /dev/vdc errs: wr 0, rd 0, flush 0, corrupt 6, gen 0
<...>
[  803.459877] BTRFS info (device vdc): relocating block group 10499391488 flags data
[  803.776810] BTRFS info (device vdc): found 29 extents, stage: move data extents
[  803.979572] BTRFS info (device vdc): found 18 extents, stage: update data pointers
[  804.276370] BTRFS info (device vdc): balance: ended with status: 0
[  804.427621] BTRFS info (device vdc): balance: start -d
[  804.454527] BTRFS info (device vdc): relocating block group 11036262400 flags data
[  804.623962] BTRFS warning (device vdc): csum failed root -9 ino 282 off 684032 csum 0x8941f998 expected csum 0x605aaa22 mirror 1
[  804.624147] BTRFS error (device vdc): bdev /dev/vdc errs: wr 0, rd 0, flush 0, corrupt 15, gen 0
[  804.624277] BTRFS warning (device vdc): csum failed root -9 ino 282 off 688128 csum 0x8941f998 expected csum 0xe90a7889 mirror 1
[  804.624435] BTRFS error (device vdc): bdev /dev/vdc errs: wr 0, rd 0, flush 0, corrupt 16, gen 0
[  804.624682] assertion failed: atomic_read(&subpage->readers) >= nbits, in fs/btrfs/subpage.c:203
[  804.624902] ------------[ cut here ]------------
[  804.624989] kernel BUG at fs/btrfs/ctree.h:3415!
cpu 0x1: Vector: 700 (Program Check) at [c000000007b47640]
    pc: c000000000af297c: assertfail.constprop.11+0x34/0x38
    lr: c000000000af2978: assertfail.constprop.11+0x30/0x38
    sp: c000000007b478e0
   msr: 800000000282b033
  current = 0xc000000007999800
  paca    = 0xc00000003fffee00	 irqmask: 0x03	 irq_happened: 0x01
    pid   = 23, comm = kworker/u4:1
kernel BUG at fs/btrfs/ctree.h:3415!
Linux version 5.12.0-rc8-00160-gcd0da6627caa (root@ltctulc6a-p1) (gcc (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.30) #25 SMP Mon May 10 01:31:44 CDT 2021
enter ? for help
[c000000007b47940] c000000000aefdac btrfs_subpage_end_reader+0x5c/0xb0
[c000000007b47980] c000000000a379f0 end_page_read+0x1d0/0x200
[c000000007b479c0] c000000000a41554 end_bio_extent_readpage+0x784/0x9b0
[c000000007b47b30] c000000000b4a234 bio_endio+0x254/0x270
[c000000007b47b70] c0000000009f6178 end_workqueue_fn+0x48/0x80
[c000000007b47ba0] c000000000a5c960 btrfs_work_helper+0x260/0x8e0
[c000000007b47c40] c00000000020a7f4 process_one_work+0x434/0x7d0
[c000000007b47d10] c00000000020ae94 worker_thread+0x304/0x570
[c000000007b47da0] c0000000002173cc kthread+0x1bc/0x1d0
[c000000007b47e10] c00000000000d6ec ret_from_kernel_thread+0x5c/0x70

-ritesh