On 2021/9/1 上午9:38, Konstantin Svist wrote:

On 8/31/21 04:05, Qu Wenruo wrote:

quoted

On 2021/8/31 下午2:25, Konstantin Svist wrote:

quoted

On 8/30/21 00:20, Qu Wenruo wrote:

quoted

On 2021/8/30 上午11:48, Konstantin Svist wrote:

quoted

I'm hoping to find several important files at this point, definitely
don't need the whole FS..

So when I run this, I get about 190 lines like

       key (256 INODE_ITEM 0) block 920748032 gen 166878
       key (52607 DIR_ITEM 988524606) block 1078902784 gen 163454
       key (52607 DIR_INDEX 18179) block 189497344 gen 30
       key (174523 INODE_REF 52607) block 185942016 gen 30
       key (361729 EXTENT_DATA 0) block 785907712 gen 166931
       key (381042 XATTR_ITEM 3817753667) block 1027391488 gen 120910

Can you provide the full output? (both stdout and stderr)

If you're concerning about the filenames, "btrfs ins dump-tree" has
--hide-names to mask all the file/dir names.

190 lines look too few than expected, thus means some tree blocks are
not read out properly.

You may want to try other bytenr to see which gives the most amount of
output (thus most possible to restore some data).

## Naming these BTR1..4
# btrfs ins dump-super -f /dev/sdb3 | grep backup_tree_root | sort -rk 4
          backup_tree_root:    787070976    gen: 166932    level: 1
### BTR1
          backup_tree_root:    786399232    gen: 166931    level: 1
### BTR2
          backup_tree_root:    781172736    gen: 166930    level: 1
### BTR3
          backup_tree_root:    778108928    gen: 166929    level: 1
### BTR4

### BTR1:
# btrfs ins dump-tree -b 787070976 --follow /dev/sdb3 | grep "(257
ROOT_ITEM" -A 5
...
     item 13 key (257 ROOT_ITEM 0) itemoff 13147 itemsize 439
          generation 166932 root_dirid 256 bytenr 786726912 level 2 refs
1      ### naming this RI1
          lastsnap 56690 byte_limit 0 bytes_used 1013104640 flags
0x0(none)
...

BTR1 -> RI1 786726912
BTR2 -> RI2 781467648
BTR3 -> RI3 780828672
BTR4 -> RI3 102760448

### inpsecting RI2
# btrfs ins dump-tree -b 781467648 --follow --bfs /dev/sdb3

quoted

RI2.inspect.stdout 2>RI2.inspect.stderr

<outputs attached>

One of the lines of this output is
          key (2334458 DIR_ITEM 3564787518) block 196816535552 gen 56498

quoted

quoted

I tried to pass these into restore, but it's not liking it:

# btrfs restore -Divf 196816535552 /dev/sdb3 .

Where the bytenr 196816535552 is from?

^^^ output from inspect RI2 -> DIR_ITEM. Probably wrong usage? :)

OK, that seems to be out of the way btrfs-restore can handle.

quoted

quoted

quoted

checksum verify failed on 786939904 wanted 0xcdcdcdcd found 0xc375d6b6
checksum verify failed on 786939904 wanted 0xcdcdcdcd found 0xc375d6b6
checksum verify failed on 786939904 wanted 0xcdcdcdcd found 0xc375d6b6
Csum didn't match
WARNING: could not setup extent tree, skipping it

This part is expected, it just tries to read extent tree which is
manually corrupted.

quoted

This is a dry-run, no files are going to be restored
Done searching

While this is not expected, as it doesn't even show any research
attempts, is the bytenr from the subtree of the subvolume 257?

Interestingly, I tried --dfs instead of --bfs and there are a lot more
entries, including filenames

BTW, thanks to the output and stderr, it shows exactly what's going
wrong.

The offending tree block, 920748032, is the first one.

If using --dfs, it will go through each child until reaches the leaves,
before going to next tree block.

And if the first child is corrupted, then it gives up immediately.

That's why I'm explicitly specifying --bfs, which will skip the
corrupted child (and its children) and go next tree blocks directly,
thus have the best chance to recovery the contents.

For the worst case, I guess you have to use "btrfs ins dump-tree" to
recovery your files, and then "btrfs-map-logical" to grab the data from
disk directly.

Meanwhile I guess I should put some time to enhance btrfs-restore to
handle the corruption you're hitting, so that we can continue to next
good tree block, without being bothered by early corrupted tree blocks.

Thanks again for looking into this!

Should I wait for a patch or is there something else I can do meanwhile?

Yes, you can use "btrfs ins dump-tree -b <bytenr> --follow --bfs" to 
dump the tree and look for the desired files. (aka, manual salvage).

The filename is included in INODE_REF key:

         item 55 key (258 INODE_REF 256) itemoff 11848 itemsize 15
                 index 3 namelen 5 name: reg_0

258 is the inode number, which can you look for its data using (<ino> 
EXTENT_DATA <offset>) key.

         item 56 key (258 EXTENT_DATA 0) itemoff 11795 itemsize 53
                 generation 19 type 1 (regular)
                 extent data disk byte 13697024 nr 4096
                 extent data offset 0 nr 4096 ram 4096
                 extent compression 0 (none)

Above (258 EXTENT_DATA 0) means, inode number 258, file data at file 
offset 0.

The logical bytenr on-disk is 13697024, size is 4096.

Then you can go "btrfs-map-logical -b 13697024" to find where the data 
is on real disks, and use dd to grab them and assemble your file.

It's time consuming and only feasible for a dozen of files.

Thanks,
Qu