On Thu, Feb 18, 2021 at 6:12 PM Daniel Dawson [off-list ref] wrote:

On 2/18/21 3:57 PM, Chris Murphy wrote:

quoted

metadata raid6 as well?

Yes.

Once everything else is figured out, you should consider converting
metadata to raid1c3.

https://lore.kernel.org/linux-btrfs/20200627032414.GX10769@hungrycats.org/ (local)

quoted

What replacement command(s) are you using?

For this drive, it was "btrfs replace start -r 3 /dev/sda3 /"

OK replace is good.

quoted

Do a RAM test for as long as you can tolerate it, or it finds the
defect. Sometimes they show up quickly, other times days.

I didn't think of a flipped bit. Thanks.

quoted

quoted

        devid    0 size 457.64GiB used 39.53GiB path /dev/sdc3
        devid    1 size 457.64GiB used 39.56GiB path /dev/sda3
        devid    2 size 457.64GiB used 39.56GiB path /dev/sdb3
        devid    4 size 457.64GiB used 39.53GiB path /dev/sdd3

This is confusing. devid 3 is claimed to be missing, but fi show isn't
showing any missing devices. If none of sd[abcd] are devid 3, then
what dev node is devid 3 and where is it?

It looks to me like btrfs is temporarily assigning devid 0 to the new
device being used as a replacement.That is what I observed before; once
the replace operation was complete, it went back to the normal number.
Since the replacement didn't finish this time, sdc3 is still devid 0.

The new replacement is devid 0 during the replacement. The drive being
replaced keeps its devid until the end, and then there's a switch,
that device is removed, and the signature on the old drive is wiped.
Sooo.... something is still wrong with the above because there's no
devid 3, there's kernel and btrfs check messages saying devid 3 is
missing.

It doesn't seem likely that /dev/sdc3 is devid 3 because it can't be
both missing and be the mounted dev node.

[  202.676601] BTRFS warning (device sdc3): devid 3 uuid 911a642e-0a4c-4483-9a1f-cde7b87c5519 is missing

Try a reboot, and use blkid to check you've got all devices + 1 (the
new one that failed replacement). Verify all supers as well with
'btrfs rescue super-recover -v' and that it all correlates with 'btrfs
filesystem show' as well.

What should be true is the replace will resume upon being normally
mounted. But for that to happen, all the drives + 1 must be available.

If a tree log is damaged and prevents mount then, you need to make a
calculation. You can try to mount with ro,nologreplay and freshen
backups for anything you'd rather not lose - just in case things get
worse. And then you can zero the log and see if that'll let you
normally mount the device (i.e. rw and not degraded). But some of it
will depend on what's wrong.



-- 
Chris Murphy