Re: All files are damaged after btrfs restore

On Tue, Mar 9, 2021 at 10:03 AM Sebastian Roller
[off-list ref] wrote:

I found 12 of these 'tree roots' on the volume. All the snapshots are
under the same tree root. This seems to be the subvolume where I put
the snapshots.

Snapshots are subvolumes. All of them will appear in the root tree,
even if they're organized as being in a directory or in some other
subvolume.

So for the snapshots there is only one option to use
with btrfs restore -r.

It can be done by its own root node address using -f or by subvolid
using -r. The latter needs to be looked up in a reliable root tree.
But I think the distinction may not matter here because really it's
the chunk tree that's messed up, and that's what's used to find
everything. The addresses in the file tree (the subvolume/snapshot
tree that contains file listings, inodes, metadata, and the address of
the file) are all logical addresses in btrfs linear space. That means
nothing without the translation to physical device and blocks, which
is the job of the chunk tree.

But I also found the data I'm looking for under
some other of these tree roots. One of them is clearly the subvolume
the backup went to (the source of the snapshots). But there is also a
very old snapshot (4 years old) that has a tree root on its own. The
files I restored  from there are different -- regarding checksums.
They are also corrupted, but different. I have to do some more
hexdumps to figure out, if it's better.

Unfortunately when things are messed up badly, the recovery tools may
be looking at a wrong or partial checksum tree and it just spits out
checksum complaints as a matter of course. You'd have to inspect the
file contents themselves, the checksum warnings might be real or
bogus.

quoted

OK this is interesting. There's two chunk trees to choose from. So is
the restore problem because older roots point to the older chunk tree
which is already going stale, and just isn't assembling blocks
correctly anymore? Or is it because the new chunk tree is bad?

Is there a way to choose the chunk tree I'm using for operations like
btrfs restore?

Looks like the answer is no. The chunk tree really has to be correct
first before anything else because it's central to doing all the
logical to physical address translation. And if it's busted and can't
be repaired then nothing else is likely to work or be repairable. It's
that critical.

I already ran chunk-recover. It needs two days to finish. But I used
btrfs-tools version 4.14 and it failed.

I'd have to go dig in git history to even know if there's been
improvements in chunk recover since then. But I pretty much consider
any file system's tool obsolete within a year. I think it's total
nonsense that distributions are intentionally using old tools.

root@hikitty:/mnt$ btrfs rescue chunk-recover /dev/sdf1
Scanning: DONE in dev0
checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
checksum verify failed on 99593231630336 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
checksum verify failed on 124762809384960 found E4E3BDB6 wanted 00000000
bytenr mismatch, want=124762809384960, have=0
open with broken chunk error
Chunk tree recovery failed

I could try again with a newer version. (?) Because with version 4.14
also btrfs restore failed.

It is entirely possible that 5.11 fails exactly the same way because
it's just too badly damaged for the current state of the recovery
tools to deal with damage of this kind. But it's also possible it'll
work. It's a coin toss unless someone else a lot more familiar with
the restore code speaks up. But looking at just the summary change
log, it looks like no work has happened in chunk recover for a while.

https://btrfs.wiki.kernel.org/index.php/Changelog

quoted

btrfs insp dump-t -t 1 /dev/sdi1

And you'll need to look for a snapshot name in there, find its bytenr,
and let's first see if just using that works. If it doesn't then maybe
combining it with the next most recent root tree will work.

I am working backwards right now using btrfs restore -f in combination
with -t. So far no success.

Yep. I think it comes down to the chunk tree needing to be reasonable
first, before anything else is possible.


-- 
Chris Murphy