On Mon 01 Feb 2021 at 10:35, Qu Wenruo [off-list ref] 
wrote:

On 2021/1/29 下午2:39, Erik Jensen wrote:

quoted

On Mon, Jan 25, 2021 at 8:54 PM Erik Jensen 
[off-list ref] wrote:

quoted

On Wed, Jan 20, 2021 at 1:08 AM Erik Jensen 
[off-list ref] wrote:

quoted

On Wed, Jan 20, 2021 at 12:31 AM Qu Wenruo 
[off-list ref] wrote:

quoted

On 2021/1/20 下午4:21, Qu Wenruo wrote:

quoted

On 2021/1/19 下午5:28, Erik Jensen wrote:

quoted

On Mon, Jan 18, 2021 at 9:22 PM Erik Jensen 
[off-list ref]
wrote:

quoted

On Mon, Jan 18, 2021 at 4:12 AM Erik Jensen 
[off-list ref]
wrote:

quoted

The offending system is indeed ARMv7 (specifically a 
Marvell ARMADA®
388), but I believe the Broadcom BCM2835 in my Raspberry 
Pi is
actually ARMv6 (with hardware float support).

Using NBD, I have verified that I receive the same error 
when
attempting to mount the filesystem on my ARMv6 Raspberry 
Pi:
[ 3491.339572] BTRFS info (device dm-4): disk space 
caching is enabled
[ 3491.394584] BTRFS info (device dm-4): has skinny 
extents
[ 3492.385095] BTRFS error (device dm-4): bad tree block 
start, want
26207780683776 have 3395945502747707095
[ 3492.514071] BTRFS error (device dm-4): bad tree block 
start, want
26207780683776 have 3395945502747707095
[ 3492.553599] BTRFS warning (device dm-4): failed to 
read tree root
[ 3492.865368] BTRFS error (device dm-4): open_ctree 
failed

The Raspberry Pi is running Linux 5.4.83.

Okay, after some more testing, ARM seems to be irrelevant, 
and 32-bit
is the key factor. On a whim, I booted up an i686, 5.8.14 
kernel in a
VM, attached the drives via NBD, ran cryptsetup, tried to 
mount, and…
I got the exact same error message.

My educated guess is on 32bit platforms, we passed 
incorrect sector into
bio, thus gave us garbage.

To prove that, you can use bcc tool to verify it.
biosnoop can do that:
https://github.com/iovisor/bcc/blob/master/tools/biosnoop_example.txt

Just try mount the fs with biosnoop running.
With "btrfs ins dump-tree -t chunk <dev>", we can manually 
calculate the
offset of each read to see if they matches.
If not match, it would prove my assumption and give us a 
pretty good
clue to fix.

Thanks,
Qu

quoted

Is this bug happening only on the fs, or any other btrfs 
can also
trigger similar problems on 32bit platforms?

Thanks,
Qu

I have only observed this error on this file system. 
Additionally, the
error mounting with the NAS only started after I did a `btrfs 
replace`
on all five 8TB drives using an x86_64 system. (Ironically, I 
did this
with the goal of making it faster to use the filesystem on 
the NAS by
re-encrypting the drives to use a cipher supported by my 
NAS's crypto
accelerator.)

Maybe this process of shuffling 40TB around caused some value 
in the
filesystem to increment to the point that a calculation using 
it
overflows on 32-bit systems?

I should be able to try biosnoop later this week, and I'll 
report back
with the results.

Okay, I tried running biosnoop, but I seem to be running into 
this
bug: https://github.com/iovisor/bcc/issues/3241 (That bug was 
reported
for cpudist, but I'm seeing the same error when I try to run
biosnoop.)

Anything else I can try?

Is it possible to add printks to retrieve the same data?

Sorry for the late reply, busying testing subpage patchset. (And
unfortunately no much process).

If bcc is not possible, you can still use ftrace events, but
unfortunately I didn't find good enough one. (In fact, the trace 
events
for block layer is pretty limited).

You can try to add printk()s in function blk_account_io_done() 
to
emulate what's done in function trace_req_completion() of 
biosnoop.

The time delta is not important, we only need the device name, 
sector
and length.

Tips: There are ftrace events called block:block_rq_issue and
block:block_rq_complete to fetch those infomation. No need to
add printk().

Thanks,
Qu