On Tue, Feb 07, 2012 at 12:17:23PM +0800, Liu Bo wrote:

quoted

quoted

The failure occurred while the volumes were online and in use, so in
addition to what was unreadable, all pending writes to the device
between the failure and when the problem was discovered were lost as
well.

Hi Ryan,

So on the failure, what does dmesg show?  checksum errors?

Dmesg at the time showed block errors on the RAID due to the multi-disk
failure.  I do have a log from that time which I have attached, including btrfs
unhappiness at the time.

Here is the oops I currently get on 3.2.2 when trying to mount the btrfs volume
of the two that btrfs-show is able to detect:

[ 1023.151683] device label vicep-library devid 1 transid 575931 /dev/mapper/tr5ut-vicep--library
[ 1023.152136] btrfs: use lzo compression
[ 1023.152174] btrfs: disk space caching is enabled
[ 1023.191409] btrfs: dm-32 checksum verify failed on 317874630656 wanted 28ABE8A6 found 8E19212D level 0
[ 1023.211750] btrfs: dm-32 checksum verify failed on 317874630656 wanted 28ABE8A6 found 491D9C1A level 0
[ 1023.216243] btrfs: dm-32 checksum verify failed on 317874630656 wanted 28ABE8A6 found 8E19212D level 0
[ 1023.224252] btrfs: dm-32 checksum verify failed on 317874630656 wanted 28ABE8A6 found 491D9C1A level 0
[ 1023.224521] btrfs: dm-32 checksum verify failed on 317874630656 wanted 28ABE8A6 found 491D9C1A level 0
[ 1023.232211] btrfs: dm-32 checksum verify failed on 317874630656 wanted 28ABE8A6 found 8E19212D level 0
[ 1023.232456] btrfs: dm-32 checksum verify failed on 317874630656 wanted 28ABE8A6 found 491D9C1A level 0
[ 1023.232549] ------------[ cut here ]------------
[ 1023.232591] kernel BUG at fs/btrfs/disk-io.c:1203!
[ 1023.232627] invalid opcode: 0000 [#1] SMP
[ 1023.232723] CPU 1
[ 1023.232755] Modules linked in: ext2 ext4 jbd2 crc16 it87 hwmon_vid loop snd_hda_codec_hdmi tpm_tis tpm tpm_bios snd_hda_codec_realtek pcspkr evdev wmi snd_hda_intel i2c_piix4 i2c_core k8temp edac_core e
dac_mce_amd snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc shpchp processor button thermal_sys pci_hotplug parport_pc parport ext3 jbd mbcache dm_snapshot aes_x86_64 aes_generic dm_crypt dm_mod raid1 md_mod nbd btrfs zlib_deflate crc32c libcrc32c xts gf128mul sg sr_mod cdrom sd_mod crc_t10dif ata_generic ohci_hcd pata_atiixp firewire_ohci ahci libahci firewire_core ehci_hcd libata tulip crc_itu_t scsi_mod usbcore floppy r8169 mii usb_common [last unloaded: scsi_wait_scan]
[ 1023.235168]
[ 1023.235203] Pid: 4829, comm: mount Not tainted 3.2.2 #3 Gigabyte Technology Co., Ltd. GA-MA78GPM-DS2H/GA-MA78GPM-DS2H
[ 1023.235335] RIP: 0010:[<ffffffffa00c0a9a>]  [<ffffffffa00c0a9a>] find_and_setup_root+0x5c/0xdc [btrfs]
[ 1023.235437] RSP: 0018:ffff8801a7607b98  EFLAGS: 00010282
[ 1023.235473] RAX: 00000000fffffffe RBX: ffff8801a798b800 RCX: 0000000000000005
[ 1023.235510] RDX: 00000000fffffffb RSI: 000000000001af60 RDI: ffffea00069b1d40
[ 1023.235547] RBP: ffff8801a798f800 R08: ffffffffa00bc092 R09: 0000000000000000
[ 1023.235584] R10: ffff8801a798f800 R11: 0000000000000000 R12: 0000000000000002
[ 1023.235621] R13: ffff8801a7989400 R14: 000000000008c9bb R15: ffff8801a772f718
[ 1023.235659] FS:  00007fee836557e0(0000) GS:ffff8801afc40000(0000) knlGS:0000000000000000
[ 1023.235699] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1023.235735] CR2: 00007fee836a4000 CR3: 00000001a7f41000 CR4: 00000000000006e0
[ 1023.235772] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1023.235809] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1023.235846] Process mount (pid: 4829, threadinfo ffff8801a7606000, task ffff8801a7f521c0)
[ 1023.235886] Stack:
[ 1023.235920]  0000000000000002 000000000008c9bb ffff8801a772f000 ffff8801a798f800
[ 1023.236080]  ffff8801a772d000 ffffffffa00c4436 0000000000000003 ffffffff817b16e0
[ 1023.236240]  0000000000001000 00001000811a57f0 ffff8801a7989680 ffff8801a798b800
[ 1023.236400] Call Trace:
[ 1023.236448]  [<ffffffffa00c4436>] ? open_ctree+0xf6c/0x1535 [btrfs]
[ 1023.236489]  [<ffffffff810ff823>] ? sget+0x39a/0x3ac
[ 1023.236501]  [<ffffffffa00a9fb5>] ? btrfs_mount+0x3a2/0x539 [btrfs]
[ 1023.236501]  [<ffffffff810d54ed>] ? pcpu_next_pop+0x37/0x43
[ 1023.236501]  [<ffffffff810d50f3>] ? cpumask_next+0x18/0x1a
[ 1023.236501]  [<ffffffff810d6502>] ? pcpu_alloc+0x875/0x8be
[ 1023.236501]  [<ffffffff810ff3ab>] ? mount_fs+0x6c/0x14a
[ 1023.236501]  [<ffffffff81113715>] ? vfs_kern_mount+0x61/0x97
[ 1023.236501]  [<ffffffff81114a2c>] ? do_kern_mount+0x49/0xd6
[ 1023.236501]  [<ffffffff811151e1>] ? do_mount+0x728/0x792
[ 1023.236501]  [<ffffffff810ee478>] ? alloc_pages_current+0xa7/0xc9
[ 1023.236501]  [<ffffffff811152d3>] ? sys_mount+0x88/0xc3
[ 1023.236501]  [<ffffffff81341152>] ? system_call_fastpath+0x16/0x1b
[ 1023.236501] Code: 24 24 e8 23 f5 ff ff 48 8d 53 20 48 8d 8b 0f 01 00 00 4c 89 e6 48 89 ef e8 fc b4 ff ff 89 c2 b8 fe ff ff ff 83 fa 00 7f 79 74 04 <0f> 0b eb fe 80 bb 0e 01 00 00 00 48 8b ab c0 00 00 00 75 08 8b
[ 1023.236501] RIP  [<ffffffffa00c0a9a>] find_and_setup_root+0x5c/0xdc [btrfs]
[ 1023.236501]  RSP <ffff8801a7607b98>
[ 1023.239093] ---[ end trace 5cc1f71c489542ef ]---

btrfsck is not ready for data recovery, but only for error checking.
But btrfs-tools do have some features that may help us, e.g zero-log.

More recovery details refer to the thread from Hugo:
http://www.spinics.net/lists/linux-btrfs/msg14890.html

I'll take a look at the 'restore' tool, thanks.

-- 
Ryan C. Underwood, [off-list ref]