#regzbot introduced: 90573092673c
Hi,
since upgrading from 7.1.8 to 7.2.11 a machine with root on
LVM -> bcache -> md raid5 (three SATA HDDs without TRIM) loses its root
device a few minutes after boot. The same stack ran on 7.1.x without
any bcache backing-device errors.
Storage stack
-------------
3x ST1000DM010-2EP102 (no TRIM; discard_max_hw_bytes = 0)
-> md126: raid5, 3 devices, 512K chunk, metadata 1.2
-> bcache0: md126 as backing device, cache set detached
(also reproduced while an NVMe cache was attached)
-> LVM vgpcrd5
-> btrfs (/usr, /var, /opt; mounted without any discard option)
+ ext4 (/, /home)
Kernel: 7.2.11-arch1-1 (Arch Linux), mdadm 4.6, bcache-tools 1.1
Symptoms
--------
Every boot, starting ~3 minutes after mount, dmesg fills with:
bcache: bch_count_backing_io_errors() md126: IO error on backing
device, unrecoverable
in bursts of up to ~35 lines 2 ms apart. Per boot I counted 11, 17, 22
and 63 of them. bcache's io_error_limit is 64, so once a boot crosses it
bch_cached_dev_error() stops bcache0 and root, /usr, /var and /home
disappear. No I/O errors are logged by libata, sd or md; all three
members are healthy (SMART clean, array [UUU], no rebuild).
Analysis
--------
Since 90573092673c ("md/raid5: validate discard support at request
time", v7.2-rc1) raid5_set_limits() unconditionally sets
lim.max_hw_discard_sectors = UINT_MAX;
and records member support in conf->raid5_discard_unsupported. On this
array sysfs now shows:
/sys/block/md126/queue/discard_max_hw_bytes: 2199023255040
/sys/block/md126/queue/discard_max_bytes: 2199023255040
/sys/block/md126/queue/discard_granularity: 0
/sys/block/sd[abc]/queue/discard_max_hw_bytes: 0
(md127, a raid1 on the same disks, still shows 0 for all of these)
Two upper layers react to the advertised discard support:
1. btrfs enables discard=async by default because the device advertises
discard (mount shows "ssd,discard=async" although nothing in fstab
asks for it).
2. bcache forwards discards to the backing device whenever
bdev_max_discard_sectors(dc->bdev) is non-zero (cached_dev_write()
and detached_dev_do_request()). Before 7.2 that was 0 for this array
and bcache completed the discards itself.
raid5_discard_limits() then completes every such bio with
BLK_STS_NOTSUPP. bcache's backing_request_endio() treats any non-zero
bi_status as a media error and calls bch_count_backing_io_errors(), so
each discard brings the backing device one step closer to being stopped.
With 64 small deletions on the btrfs volume a root filesystem on
raid5+bcache is gone.
The granularity is also inconsistent: discard_granularity is 0 while
discard_max_hw_bytes is non-zero, so writing any non-zero value to
discard_max_bytes fails with EINVAL (only 0 is accepted).
Related changes I found:
- efdffeb6d491 ("md: skip discard on unsupported member devices", 7.3)
fixes the same bcache symptom for raid0/raid10 via
md_submit_discard_bio(), but raid5 does not use that helper and still
returns BLK_STS_NOTSUPP in current mainline (raid5_discard_limits()).
- 85764f475f3b ("md/raid5: complete discard bios while reshape is
active") addresses a different fallout of the same commit (hang during
reshape).
I have not found a report for this raid5 + bcache case on lore.
Possible fixes (either side would do):
- md/raid5: when conf->raid5_discard_unsupported is set, complete the
bio successfully (discard is advisory) instead of BLK_STS_NOTSUPP, or
do not advertise max_hw_discard_sectors in that case, as before 7.2.
- bcache: do not count BLK_STS_NOTSUPP (or failed REQ_OP_DISCARD) in
bch_count_backing_io_errors(), the same way REQ_RAHEAD failures are
already ignored.
Workaround in use
-----------------
A udev rule writing 0 to /sys/block/md126/queue/discard_max_bytes and
/sys/block/bcache0/queue/discard_max_bytes (also inside the initramfs),
plus "nodiscard" on the btrfs mounts. With that, io_errors stays at 0.
I can test patches on this machine.
Regards,
Konstantin Nikiforov
Claude Fable 5.1