Hello, I lost access to my home BTRFS filesystem on a 4TB SATA drive without partitioning today. I am a graduate student of translation studies at a foreign university. Programming is not my specialty but have ran Linux for over a decade because I support open source endeavors. I say this for background on my technical capabilities, I am also a radio amateur extra-class in the United States. I have some technical competence and run Arch Linux for general duty. The system boots and reads the kernel into memory from an optical ROM drive, reads keyfiles from a USB stick, decrypts the home and swap partitions on different devices, asks for a password for the root partition, and finishes booting. I was rebooting my system from userspace and observed a kernel panic after finishing some work repartitioning a small USB stick using gparted. After rebooting, my home BTRFS system was not mountable. Some information seems present in some of the superblocks, but I do not understand everything. I was careful in my use of the disk tool, and it was clear that only the USB stick was being accessed, because of the simple work I was doing, resizing a partition on the obvious drive. I would like to recover the data. I finished writing my master's thesis and have already submitted it, so no real work was lost, but I would like to understand why this is happening and how the data can be recovered. A memory fault seems plausible, but the fact that I was accessin g the partitions (through gparted) seems to indicate other possibilities. Please do not get sidetracked into thinking I wrote data on the drive through carelessness. I have made many many uses of disk tools throughout my life and the simple nature of the operations which did not involve the filesystem in question are completely understandable and I remember them clearly. Please advise. Relevant /dev/fstab entry which has worked for several months until I commented it out today: #/dev/mapper/home /home btrfs rw,compress,autodefrag,inode_cache 0 1 cat /proc/version: Linux version 5.15.0-rc3-1-git-00319-g7b66f4393ad4 (linux-git@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Sun, 03 Oct 2021 17:24:26 +0000 sudo smartctl -a /dev/sdd excerpt: === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 2.5 5400 Device Model: ST4000LM024-2AN17V Serial Number: WCK03H8Y LU WWN Device Id: 5 000c50 09e55827b Firmware Version: 0001 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5526 rpm Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Oct 11 16:07:49 2021 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION ===
All Three Superblocks Damaged After Kernel Panic
Started by Kyle James Chapman on Oct 11, 2021 · 7 messages from 4 authors · Last reply Oct 12, 2021
open the first message on its own page
M Matt Corallo replied to Kyle James Chapman #
List subscribers, Kyle’s email, below, may not have made it to your inbox. VGER’s SMTP RFC violation here should have resulted in your mail server refusing to accept this. Kyle, Please note that your email client, your upstream mail server, and vger.kernel.org are all miscofigured, resulting in your email violating the SMTP RFCs upon delivery. Because individual lines are not broken and longer than 1000 characters, many folks likely didn’t receive your email (eg default exim configurations will refuse your mail). You should reach out to the administrators for sogo01.urz.uni-heidelberg.de and relay.uni-heidelberg.de and have them fix their configuration, as well as fix your local MUA. Matt
On Oct 11, 2021, at 11:09, Kyle James Chapman [off-list ref] wrote: Hello, I lost access to my home BTRFS filesystem on a 4TB SATA drive without partitioning today. I am a graduate student of translation studies at a foreign university. Programming is not my specialty but have ran Linux for over a decade because I support open source endeavors. I say this for background on my technical capabilities, I am also a radio amateur extra-class in the United States. I have some technical competence and run Arch Linux for general duty. The system boots and reads the kernel into memory from an optical ROM drive, reads keyfiles from a USB stick, decrypts the home and swap partitions on different devices, asks for a password for the root partition, and finishes booting. I was rebooting my system from userspace and observed a kernel panic after finishing some work repartitioning a small USB stick using gparted. After rebooting, my home BTRFS system was not mountable. Some information seems present in some of the superblocks, but I do not understand everything. I was careful in my use of the disk tool, and it was clear that only the USB stick was being accessed, because of the simple work I was doing, resizing a partition on the obvious drive. I would like to recover the data. I finished writing my master's thesis and have already submitted it, so no real work was lost, but I would like to understand why this is happening and how the data can be recovered. A memory fault seems plausible, but the fact that I was accessing the partitions (through gparted) seems to indicate other possibilities. Please do not get sidetracked into thinking I wrote data on the drive through carelessness. I have made many many uses of disk tools throughout my life and the simple nature of the operations which did not involve the filesystem in question are completely understandable and I remember them clearly. Please advise. Relevant /dev/fstab entry which has worked for several months until I commented it out today: #/dev/mapper/home /home btrfs rw,compress,autodefrag,inode_cache 0 1 cat /proc/version: Linux version 5.15.0-rc3-1-git-00319-g7b66f4393ad4 (linux-git@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Sun, 03 Oct 2021 17:24:26 +0000 sudo smartctl -a /dev/sdd excerpt: === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 2.5 5400 Device Model: ST4000LM024-2AN17V Serial Number: WCK03H8Y LU WWN Device Id: 5 000c50 09e55827b Firmware Version: 0001 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5526 rpm Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Oct 11 16:07:49 2021 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled
Q Qu Wenruo replied to Kyle James Chapman #
On 2021/10/11 22:13, Kyle James Chapman wrote:
Hello, I lost access to my home BTRFS filesystem on a 4TB SATA drive without partitioning today. I am a graduate student of translation studies at a foreign university. Programming is not my specialty but have ran Linux for over a decade because I support open source endeavors. I say this for background on my technical capabilities, I am also a radio amateur extra-class in the United States. I have some technical competence and run Arch Linux for general duty. The system boots and reads the kernel into memory from an optical ROM drive, reads keyfiles from a USB stick, decrypts the home and swap partitions on different devices, asks for a password for the root partition, and finishes booting. I was rebooting my system from userspace and observed a kernel panic after finishing some work repartitioning a small USB stick using gparted.
Any idea on what part is causing the panic, or any log/screenshot of the
panic message?
After rebooting, my home BTRFS system was not mountable. Some information seems present in some of the superblocks, but I do not understand everything. I was careful in my use of the disk tool, and it was clear that only the USB stick was being accessed, because of the simple work I was doing, resizing a partition on the obvious drive. I would like to recover the data. I finished writing my master's thesis and have already submitted it, so no real work was lost, but I would like to understand why this is happening and how the data can be recovered. A memory fault seems plausible, but the fact that I was accessing the partitions (through gparted) seems to indicate other possibilities. Please do not get sidetracked into thinking I wrote data on the drive through carelessness. I have made many many uses of disk tools throughout my life and the simple nature of the operations which did not involve the filesystem in question are completely understandable and I remember them clearly. Please advise. Relevant /dev/fstab entry which has worked for several months until I commented it out today: #/dev/mapper/home /home btrfs rw,compress,autodefrag,inode_cache 0 1
Inode cache is already deprecated, but it won't do anything to the
kernel, thus it shouldn't cause any problem.
cat /proc/version: Linux version 5.15.0-rc3-1-git-00319-g7b66f4393ad4 (linux-git@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Sun, 03 Oct 2021 17:24:26 +0000 sudo smartctl -a /dev/sdd excerpt: === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 2.5 5400 Device Model: ST4000LM024-2AN17V Serial Number: WCK03H8Y LU WWN Device Id: 5 000c50 09e55827b Firmware Version: 0001 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5526 rpm Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Oct 11 16:07:49 2021 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED btrfs --version: btrfs-progs v5.14.1 sudo btrfs inspect-internal dump-super -s 0 /dev/mapper/home : superblock: bytenr=65536, device=/dev/mapper/home
The first copy in fact looks pretty good.
If there is something wrong, then it would be in the backup roots or sys
chunk map.
Mind to dump the full super block with "btrfs ins dump-super -Ffa"?--------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xc59083ff [DON'T MATCH] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid bd5872ad-8fee-4d90-b048-b81120ce3254 metadata_uuid bd5872ad-8fee-4d90-b048-b81120ce3254 label generation 161054 root 4028020703232 sys_array_size 129 chunk_root_generation 154912 root_level 1 chunk_root 22020096 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 4000787030016 bytes_used 3969357348864 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x161 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA ) cache_generation 161054 uuid_tree_generation 161054 dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51 dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [match]
The 2nd one begins to have some garbage, but not yet full of garbage.
This means there are some writes into the 2nd super block, and mostly
the writes are just into part of the super block.
magic _BHRfS_M [match] fsid 4468b2e4-d249-639c-dc54-792caf170cc9 metadata_uuid 4468b2e4-d249-639c-dc54-792caf170cc9
The UUID is damaged.
label generation 161054 root 4028020703232 > sys_array_size 129 chunk_root_generation 154912 root_level 1 chunk_root 22020096 chunk_root_level 1
But then some good data.
log_root 11922660382425005768 log_root_transid 14582268926853107316 log_root_level 0 total_bytes 10532150587539085458 bytes_used 8124524553913841719
Some garbage again. I don't think it can be even caused by simple memory corruption, too many random part gets corrupted. It's really hard to say if btrfs itself is involved, until we see the calltrace. For the recovery, the superblock needs to be manually fixed to allow any further data salvage attempts. And that can only be done with the extra "btrfs ins dump-super -Ffa" output. Thanks, Qu
sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x161 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA ) cache_generation 161054 uuid_tree_generation 161054 dev_item.uuid 47593c4d-689d-4e1e-a310-dc7b8ab26c51 dev_item.fsid bd5872ad-8fee-4d90-b048-b81120ce3254 [DON'T MATCH] dev_item.type 0 dev_item.total_bytes 4000787030016 dev_item.bytes_used 3997564731392 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sudo btrfs inspect-internal dump-super -s 2 /dev/mapper/home: superblock: bytenr=274877906944, device=/dev/mapper/home --------------------------------------------------------- csum_type 62267 (INVALID) csum_size 32 csum 0x9876fd0000000000000000000000000000000000000000000000000000000000 [UNKNOWN CSUM TYPE OR SIZE] bytenr 274877906944 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match]
Thank you so much for the help! I have images of my projects and dog which passed away among other things I would like to save. Make a backup is fine to say, but I am a student and have not had much luck earning to buy a new hdd in this foreign country. Re: Any idea on what part is causing the panic, or any log/screenshot of the panic message? This remains unclear. I did not photograph the kernel panic and I have no log of the error. The two actions which I performed at the time was the execution of the reboot command and unplugging the usb stick. I then looked at the screen and saw a panic message and then hit the power switch on the power supply and reset the machine. Re: Mind to dump the full super block with "btrfs ins dump-super -Ffa"? sudo btrfs ins dump-super -Ffa /dev/mapper/home : superblock: bytenr=65536, device=/dev/mapper/home --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xc59083ff [DON'T MATCH] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid bd5872ad-8fee-4d90-b048-b81120ce3254 metadata_uuid bd5872ad-8fee-4d90-b048-b81120ce3254 label generation 161054 root 4028020703232 sys_array_size 129 chunk_root_generation 154912 root_level 1 chunk_root 22020096 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 4000787030016 bytes_used 3969357348864 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384
On 2021/10/11 22:13, Kyle James Chapman wrote:quoted
Hello, I lost access to my home BTRFS filesystem on a 4TB SATA drive without partitioning today. I am a graduate student of translation studies at a foreign university. Programming is not my specialty but have ran Linux for over a decade because I support open source endeavors. I say this for background on my technical capabilities, I am also a radio amateur extra-class in the United States. I have some technical competence and run Arch Linux for general duty. The system boots and reads the kernel into memory from an optical ROM drive, reads keyfiles from a USB stick, decrypts the home and swap partitions on different devices, asks for a password for the root partition, and finishes booting. I was rebooting my system from userspace and observed a kernel panic after finishing some work repartitioning a small USB stick using gparted.Any idea on what part is causing the panic, or any log/screenshot of the panic message?quoted
After rebooting, my home BTRFS system was not mountable. Some information seems present in some of the superblocks, but I do not understand everything. I was careful in my use of the disk tool, and it was clear that only the USB stick was being accessed, because of the simple work I was doing, resizing a partition on the obvious drive. I would like to recover the data. I finished writing my master's thesis and have already submitted it, so no real work was lost, but I would like to understand why this is happening and how the data can be recovered. A memory fault seems plausible, but the fact that I was accessing the partitions (through gparted) seems to indicate other possibilities. Please do not get sidetracked into thinking I wrote data on the drive through carelessness. I have made many many uses of disk tools throughout my life and the simple nature of the operations which did not involve the filesystem in question are completely understandable and I remember them clearly. Please advise. Relevant /dev/fstab entry which has worked for several months until I commented it out today: #/dev/mapper/home /home btrfs rw,compress,autodefrag,inode_cache 0 1Inode cache is already deprecated, but it won't do anything to the kernel, thus it shouldn't cause any problem.quoted
cat /proc/version: Linux version 5.15.0-rc3-1-git-00319-g7b66f4393ad4 (linux-git@archlinux) (gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Sun, 03 Oct 2021 17:24:26 +0000 sudo smartctl -a /dev/sdd excerpt: === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 2.5 5400 Device Model: ST4000LM024-2AN17V Serial Number: WCK03H8Y LU WWN Device Id: 5 000c50 09e55827b Firmware Version: 0001 User Capacity: 4,000,787,030,016 bytes [4.00 TB]
On 2021/10/12 15:41, K Chapman wrote:
Thank you so much for the help! I have images of my projects and dog which passed away among other things I would like to save. Make a backup is fine to say, but I am a student and have not had much luck earning to buy a new hdd in this foreign country. Re: Any idea on what part is causing the panic, or any log/screenshot of the panic message? This remains unclear. I did not photograph the kernel panic and I have no log of the error. The two actions which I performed at the time was the execution of the reboot command and unplugging the usb stick. I then looked at the screen and saw a panic message and then hit the power switch on the power supply and reset the machine.
Any clue would be helpful. Like if it's a warning or BUG_ON() or whatever elese. Another thing is, when you hit the power switch to force shutdown, does the system already hang there for a while, or you see the panic message and immediately try to force shutdown? If the power is lost at exactly certain unlucky point, I guess the SSD may have a chance to corrupt its data (which shouldn't be). For now, what we really have is just whatever left on the disk, without any clue how it's corrupted.
Re: Mind to dump the full super block with "btrfs ins dump-super -Ffa"? sudo btrfs ins dump-super -Ffa /dev/mapper/home : superblock: bytenr=65536, device=/dev/mapper/home --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xc59083ff [DON'T MATCH] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid bd5872ad-8fee-4d90-b048-b81120ce3254 metadata_uuid bd5872ad-8fee-4d90-b048-b81120ce3254 label generation 161054 root 4028020703232 sys_array_size 129 chunk_root_generation 154912 root_level 1 chunk_root 22020096 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 4000787030016 bytes_used 3969357348864 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x161 ( MIXED_BACKREF |
So far, the first super block looks completely fine. I guess the corrupted part is in the unused part of the sys chunk array. So you can re-calculte the checksum of the first super block, and try mount/btrfs-rescue. If you don't have the tool to do it by your own, please send the 4K super block (dd if=/dev/mapper/home bs=1 skip=65536 count=4096 of=/tmp/sb.dump) to me so I could do that for you. But considering how corrupted other super blocks are, I doubt the chance to mount. Thanks, Qu
superblock: bytenr=67108864, device=/dev/mapper/home --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x65f1ab31 [DON'T MATCH] bytenr 14039944498490823899 flags 0xcdc898509422934d ( WRITTEN | unknown flag: 0xcdc898509422934c ) magic _BHRfS_M [match] fsid 4468b2e4-d249-639c-dc54-792caf170cc9 metadata_uuid 4468b2e4-d249-639c-dc54-792caf170cc9 label generation 161054 root 4028020703232 sys_array_size 129 chunk_root_generation 154912 root_level 1 chunk_root 22020096 chunk_root_level 1 log_root 11922660382425005768 log_root_transid 14582268926853107316 log_root_level 0 total_bytes 10532150587539085458 bytes_used 8124524553913841719 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x161 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA )
Qu, RE: If you don't have the tool to do it by your own, please send the 4K super block (dd if=/dev/mapper/home bs=1 skip=65536 count=4096 of=/tmp/sb.dump) to me so I could do that for you. See attached. RE: Power loss and data writes. I believe the possibility is present but remote. I have not had a panic on x86 in many years during general work, this is more of an indication of some sort of problem than simply removing power at a bad time. It seems more likely, that for whatever reason, the kernel wrote spurious data to certain sectors, it will be interesting to see. Thank you! Kyle C.
On 2021/10/12 18:35, K Chapman wrote:
Qu, RE: If you don't have the tool to do it by your own, please send the 4K super block (dd if=/dev/mapper/home bs=1 skip=65536 count=4096 of=/tmp/sb.dump) to me so I could do that for you. See attached.
Re-csumed super block is attached. It passes my local test using btrfs-ins-dump-super. Thus the csum should be correct. Wish you good luck salvaging your data. I would try mount with "-o ro,rescue=all" first, if no luck then btrfs-restore --dry-run to see what can be salvaged. But I doubt how many can be salvaged, consider how corrupted the backup super blocks are. Thanks, Qu
RE: Power loss and data writes. I believe the possibility is present but remote. I have not had a panic on x86 in many years during general work, this is more of an indication of some sort of problem than simply removing power at a bad time. It seems more likely, that for whatever reason, the kernel wrote spurious data to certain sectors, it will be interesting to see. Thank you! Kyle C.