Dear devs,
since 26.07. scrub keeps reporting csum errors with random files.
I replaced these files from backups. Then deleted the snapshots that still contained the
the corrupt files. Snapshot with corrupt files I have determined with md5sum, here I get an input/output error.
Following new scrub, still finds new csum errors that did not exist before.
Beginning with Kernel 5.10.52, current 5.10.55
btrfs-progs 5.13
Disk layout with problems:
mdadm raid10 4xhdd => bcache => luks
mdadm raid6 4xhdd => bcache => luks
Already replaced 2 old hdds with high Raw_Read_Error_Rate values.
Aug 02 15:43:18 server kernel: BTRFS info (device dm-0): scrub: started on devid 1
Aug 02 15:46:06 server kernel: BTRFS warning (device dm-0): checksum error at logical 462380818432 on dev /dev/mapper/root, physical 31640150016, root 29539, inode 27412268, offset 131072, length 4096, links 1 (path: docker-volumes/mayan-edms/media/document_cache/804391c5-e3fe-4941-96dc-ecc0a1d5d8c9-23-1815-92bcac02c4a72586e21044c0b244b052f5747c7d2c25e6086ca89ca64098e3f3)
Aug 02 15:46:06 server kernel: BTRFS error (device dm-0): bdev /dev/mapper/root errs: wr 0, rd 0, flush 0, corrupt 414, gen 0
Aug 02 15:46:06 server kernel: BTRFS error (device dm-0): unable to fixup (regular) error at logical 462380818432 on dev /dev/mapper/root
Aug 02 15:47:25 server kernel: BTRFS info (device dm-0): scrub: finished on devid 1 with status: 0
On Mon, Aug 02, 2021 at 04:20:43PM +0200, telsch wrote:
Dear devs,
since 26.07. scrub keeps reporting csum errors with random files.
I replaced these files from backups. Then deleted the snapshots that still contained the
the corrupt files. Snapshot with corrupt files I have determined with md5sum, here I get an input/output error.
Following new scrub, still finds new csum errors that did not exist before.
Beginning with Kernel 5.10.52, current 5.10.55
btrfs-progs 5.13
Disk layout with problems:
mdadm raid10 4xhdd => bcache => luks
mdadm raid6 4xhdd => bcache => luks
Missing information: what are the model/firmware revision of the
devices, is the bcache in writeback or writethrough mode, how many
SSDs are there, is there a separate bcache SSD for each HDD or are
multiple HDDs sharing any bcache SSDs?
Based on the symptoms, the most likely case is there's one SSD or a
mdadm-mirrored pair of SSDs for bcache, and at least one SSD is failing.
It may be a SSD that is not rated for caching use cases, or a SSD with
firmware bugs that prevent reliable error reporting. It's also possible
one or more HDDs is silently corrupting data, but that is less common
in the wild.
The writeback/writethrough question informs us how recoverable the
damage is. Damage in writethrough mode is recoverable in some cases
by simply removing the cache and mounting the backing drives directly.
In writeback mode the data is already gone, and if the SSD fails before
the bcache can be fully flushed, the filesystem will be destroyed.
Already replaced 2 old hdds with high Raw_Read_Error_Rate values.
1. Replace all SSDs in the system, or cleanly remove the SSD devices
from the bcache. Silent corruption is a common early failure mode on
SSDs, and bcache doesn't use checksums to detect it. If you continue
to use bcache in writeback mode with a bad SSD, it will corrupt more
and more data until the SSD finally dies, and the filesystem will be
unrecoverable after that. If you're using bcache in writethrough mode,
the corruption will only be affecting reads, and you can simply remove
and discard the SSD without damaging the filesystem (it might even fix
previously uncorrectable data if the copy on the backing HDDs is intact).
2. If that doesn't solve the problem, run mdadm checkarray and look at
/sys/block/md*/md/mismatch_cnt afterwards. checkarray doesn't report
non-zero mismatch_cnt, so you'll need to check for it separately.
If the mismatch_cnt is non-zero, you'll have to figure out which
drive is at fault somehow. Neither mdadm nor SMART will tell you if
one drive's cache RAM goes bad in an array: mdadm doesn't know which
drive is correct when they have different contents, and generally SMART
cannot detect failures inside the disk's firmware runtime environment
that might affect data integrity like cache DRAM failure. You might
be able to identify the bad drive by manually inspecting blocks with
different data, but there's no automated way to do this.
3. To avoid future problems, break the mdadm arrays into separate
devices and put them all in a btrfs raid1 so in future btrfs can tell you
immediately which device is corrupting your data. (raid1 here to avoid
issues with striped access through a SSD cache). This might be tricky
to achieve before the bad device is identified, because the bad device
will keep injecting corrupted data that will abort btrfs resize/device
delete operations.
Aug 02 15:43:18 server kernel: BTRFS info (device dm-0): scrub: started on devid 1
Aug 02 15:46:06 server kernel: BTRFS warning (device dm-0): checksum error at logical 462380818432 on dev /dev/mapper/root, physical 31640150016, root 29539, inode 27412268, offset 131072, length 4096, links 1 (path: docker-volumes/mayan-edms/media/document_cache/804391c5-e3fe-4941-96dc-ecc0a1d5d8c9-23-1815-92bcac02c4a72586e21044c0b244b052f5747c7d2c25e6086ca89ca64098e3f3)
Aug 02 15:46:06 server kernel: BTRFS error (device dm-0): bdev /dev/mapper/root errs: wr 0, rd 0, flush 0, corrupt 414, gen 0
Aug 02 15:46:06 server kernel: BTRFS error (device dm-0): unable to fixup (regular) error at logical 462380818432 on dev /dev/mapper/root
Aug 02 15:47:25 server kernel: BTRFS info (device dm-0): scrub: finished on devid 1 with status: 0
On Mon, Aug 02, 2021 at 04:20:43PM +0200, telsch wrote:
quoted
Dear devs,
since 26.07. scrub keeps reporting csum errors with random files.
I replaced these files from backups. Then deleted the snapshots that still contained the
the corrupt files. Snapshot with corrupt files I have determined with md5sum, here I get an input/output error.
Following new scrub, still finds new csum errors that did not exist before.
Beginning with Kernel 5.10.52, current 5.10.55
btrfs-progs 5.13
Disk layout with problems:
mdadm raid10 4xhdd => bcache => luks
mdadm raid6 4xhdd => bcache => luks
Missing information: what are the model/firmware revision of the
devices, is the bcache in writeback or writethrough mode, how many
SSDs are there, is there a separate bcache SSD for each HDD or are
multiple HDDs sharing any bcache SSDs?
1 SanDisk SDSSDA120G/Firmware Version: Z22000RL
I'm using only one SSD in writearound mode for both arrays.
Based on the symptoms, the most likely case is there's one SSD or a
mdadm-mirrored pair of SSDs for bcache, and at least one SSD is failing.
It may be a SSD that is not rated for caching use cases, or a SSD with
firmware bugs that prevent reliable error reporting. It's also possible
one or more HDDs is silently corrupting data, but that is less common
in the wild.
The writeback/writethrough question informs us how recoverable the
damage is. Damage in writethrough mode is recoverable in some cases
by simply removing the cache and mounting the backing drives directly.
In writeback mode the data is already gone, and if the SSD fails before
the bcache can be fully flushed, the filesystem will be destroyed.
quoted
Already replaced 2 old hdds with high Raw_Read_Error_Rate values.
1. Replace all SSDs in the system, or cleanly remove the SSD devices
from the bcache. Silent corruption is a common early failure mode on
SSDs, and bcache doesn't use checksums to detect it. If you continue
to use bcache in writeback mode with a bad SSD, it will corrupt more
and more data until the SSD finally dies, and the filesystem will be
unrecoverable after that. If you're using bcache in writethrough mode,
the corruption will only be affecting reads, and you can simply remove
and discard the SSD without damaging the filesystem (it might even fix
previously uncorrectable data if the copy on the backing HDDs is intact).
Thanks for your explanations!
Since I am in writearound mode and the files that are corrupted were not
rewritten, I had not thought about a failing SSD and corrupted bcache reads.
As last step I detached the caching device, and the previous input/output
errors disapperd :) So you was right, the SSD looks faulty. Many thanks for
your help!
2. If that doesn't solve the problem, run mdadm checkarray and look at
/sys/block/md*/md/mismatch_cnt afterwards. checkarray doesn't report
non-zero mismatch_cnt, so you'll need to check for it separately.
If the mismatch_cnt is non-zero, you'll have to figure out which
drive is at fault somehow. Neither mdadm nor SMART will tell you if
one drive's cache RAM goes bad in an array: mdadm doesn't know which
drive is correct when they have different contents, and generally SMART
cannot detect failures inside the disk's firmware runtime environment
that might affect data integrity like cache DRAM failure. You might
be able to identify the bad drive by manually inspecting blocks with
different data, but there's no automated way to do this.
It seems Arch Linux does not provide the checkarray script, so i run the
check manually - mismatch_cnt is still zero after.
3. To avoid future problems, break the mdadm arrays into separate
devices and put them all in a btrfs raid1 so in future btrfs can tell you
immediately which device is corrupting your data. (raid1 here to avoid
issues with striped access through a SSD cache). This might be tricky
to achieve before the bad device is identified, because the bad device
will keep injecting corrupted data that will abort btrfs resize/device
delete operations.
On new systems i have already used btrfs raid1 instead of mdadm.
quoted
Aug 02 15:43:18 server kernel: BTRFS info (device dm-0): scrub: started on devid 1
Aug 02 15:46:06 server kernel: BTRFS warning (device dm-0): checksum error at logical 462380818432 on dev /dev/mapper/root, physical 31640150016, root 29539, inode 27412268, offset 131072, length 4096, links 1 (path: docker-volumes/mayan-edms/media/document_cache/804391c5-e3fe-4941-96dc-ecc0a1d5d8c9-23-1815-92bcac02c4a72586e21044c0b244b052f5747c7d2c25e6086ca89ca64098e3f3)
Aug 02 15:46:06 server kernel: BTRFS error (device dm-0): bdev /dev/mapper/root errs: wr 0, rd 0, flush 0, corrupt 414, gen 0
Aug 02 15:46:06 server kernel: BTRFS error (device dm-0): unable to fixup (regular) error at logical 462380818432 on dev /dev/mapper/root
Aug 02 15:47:25 server kernel: BTRFS info (device dm-0): scrub: finished on devid 1 with status: 0
On Tue, Aug 03, 2021 at 08:55:07PM +0200, telsch wrote:
quoted
On Mon, Aug 02, 2021 at 04:20:43PM +0200, telsch wrote:
quoted
Dear devs,
since 26.07. scrub keeps reporting csum errors with random files.
I replaced these files from backups. Then deleted the snapshots that still contained the
the corrupt files. Snapshot with corrupt files I have determined with md5sum, here I get an input/output error.
Following new scrub, still finds new csum errors that did not exist before.
Beginning with Kernel 5.10.52, current 5.10.55
btrfs-progs 5.13
Disk layout with problems:
mdadm raid10 4xhdd => bcache => luks
mdadm raid6 4xhdd => bcache => luks
Missing information: what are the model/firmware revision of the
devices, is the bcache in writeback or writethrough mode, how many
SSDs are there, is there a separate bcache SSD for each HDD or are
multiple HDDs sharing any bcache SSDs?
1 SanDisk SDSSDA120G/Firmware Version: Z22000RL
I have some of those! I can confirm they silently corrupt data as they
approach the end of their very short lives. I've seen no evidence that
the firmware is able to detect or report any errors before the drive dies,
despite providing drives many opportunities to do so as they died.
It's the worst model in a 2.5" SATA SSD form factor that I've ever tested,
setting a benchmark for failure that remains uncontested to this day.
Kingston made some terrible models that have similar firmware bugs,
but quantitatively they were still much better than SanDisk: lower
device AFR, longer mean time to first host detected corruption.
I'm using only one SSD in writearound mode for both arrays.
I tested mine in a similar setup. As long as you remove the SSD early
enough, it should be recoverable...
quoted
Based on the symptoms, the most likely case is there's one SSD or a
mdadm-mirrored pair of SSDs for bcache, and at least one SSD is failing.
It may be a SSD that is not rated for caching use cases, or a SSD with
firmware bugs that prevent reliable error reporting. It's also possible
one or more HDDs is silently corrupting data, but that is less common
in the wild.
The writeback/writethrough question informs us how recoverable the
damage is. Damage in writethrough mode is recoverable in some cases
by simply removing the cache and mounting the backing drives directly.
In writeback mode the data is already gone, and if the SSD fails before
the bcache can be fully flushed, the filesystem will be destroyed.
quoted
Already replaced 2 old hdds with high Raw_Read_Error_Rate values.
1. Replace all SSDs in the system, or cleanly remove the SSD devices
from the bcache. Silent corruption is a common early failure mode on
SSDs, and bcache doesn't use checksums to detect it. If you continue
to use bcache in writeback mode with a bad SSD, it will corrupt more
and more data until the SSD finally dies, and the filesystem will be
unrecoverable after that. If you're using bcache in writethrough mode,
the corruption will only be affecting reads, and you can simply remove
and discard the SSD without damaging the filesystem (it might even fix
previously uncorrectable data if the copy on the backing HDDs is intact).
Thanks for your explanations!
Since I am in writearound mode and the files that are corrupted were not
rewritten, I had not thought about a failing SSD and corrupted bcache reads.
As last step I detached the caching device, and the previous input/output
errors disapperd :) So you was right, the SSD looks faulty. Many thanks for
your help!
...and it is. Success! \o/
quoted
2. If that doesn't solve the problem, run mdadm checkarray and look at
/sys/block/md*/md/mismatch_cnt afterwards. checkarray doesn't report
non-zero mismatch_cnt, so you'll need to check for it separately.
If the mismatch_cnt is non-zero, you'll have to figure out which
drive is at fault somehow. Neither mdadm nor SMART will tell you if
one drive's cache RAM goes bad in an array: mdadm doesn't know which
drive is correct when they have different contents, and generally SMART
cannot detect failures inside the disk's firmware runtime environment
that might affect data integrity like cache DRAM failure. You might
be able to identify the bad drive by manually inspecting blocks with
different data, but there's no automated way to do this.
It seems Arch Linux does not provide the checkarray script, so i run the
check manually - mismatch_cnt is still zero after.
quoted
3. To avoid future problems, break the mdadm arrays into separate
devices and put them all in a btrfs raid1 so in future btrfs can tell you
immediately which device is corrupting your data. (raid1 here to avoid
issues with striped access through a SSD cache). This might be tricky
to achieve before the bad device is identified, because the bad device
will keep injecting corrupted data that will abort btrfs resize/device
delete operations.
On new systems i have already used btrfs raid1 instead of mdadm.
quoted
quoted
Aug 02 15:43:18 server kernel: BTRFS info (device dm-0): scrub: started on devid 1
Aug 02 15:46:06 server kernel: BTRFS warning (device dm-0): checksum error at logical 462380818432 on dev /dev/mapper/root, physical 31640150016, root 29539, inode 27412268, offset 131072, length 4096, links 1 (path: docker-volumes/mayan-edms/media/document_cache/804391c5-e3fe-4941-96dc-ecc0a1d5d8c9-23-1815-92bcac02c4a72586e21044c0b244b052f5747c7d2c25e6086ca89ca64098e3f3)
Aug 02 15:46:06 server kernel: BTRFS error (device dm-0): bdev /dev/mapper/root errs: wr 0, rd 0, flush 0, corrupt 414, gen 0
Aug 02 15:46:06 server kernel: BTRFS error (device dm-0): unable to fixup (regular) error at logical 462380818432 on dev /dev/mapper/root
Aug 02 15:47:25 server kernel: BTRFS info (device dm-0): scrub: finished on devid 1 with status: 0
On 04.08.2021 00:16, Zygo Blaxell wrote:
quoted
1 SanDisk SDSSDA120G/Firmware Version: Z22000RL
I have some of those! I can confirm they silently corrupt data as they
approach the end of their very short lives. I've seen no evidence that
the firmware is able to detect or report any errors before the drive dies,
despite providing drives many opportunities to do so as they died.
What about adding them to
https://btrfs.wiki.kernel.org/index.php/Hardware_bugs?