Re: Large multi-device BTRFS array (usually) fails to mount on boot.

February 3, 2021 4:56 PM, "Qu Wenruo" [off-list ref] wrote:

On 2021/2/4 上午5:54, joshua@mailmag.net wrote:

quoted

Good Evening.

I have a large BTRFS array, (14 Drives, ~100 TB RAW) which has been having problems mounting on
boot without timing out. This causes the system to drop to emergency mode. I am then able to mount
the array in emergency mode and all data appears fine, but upon reboot it fails again.

I actually first had this problem around a year ago, and initially put considerable effort into
extending the timeout in systemd, as I believed that to be the problem. However, all the methods I
attempted did not work properly or caused the system to continue booting before the array was
mounted, causing all sorts of issues. Eventually, I was able to almost completely resolve it by
defragmenting the extent tree and subvolume tree for each subvolume. (btrfs fi defrag
/mountpoint/subvolume/) This seemed to reduce the time required to mount, and made it mount on boot
the majority of the time.

Recently I expanded the array yet again by adding another drive, (and some more data) and now I am
having the same issue again. I've posted the relevant entries from my dmesg, as well as some
information on my array and system below. I ran a defrag as mentioned above on each subvolume, and
was able to get the system to boot successfully. Any ideas on a more reliable and permanent
solution this this? Thanks much!

dmesg entries upon boot:
[ 22.775439] BTRFS info (device sdh): use lzo compression, level 0
[ 22.775441] BTRFS info (device sdh): using free space tree
[ 22.775442] BTRFS info (device sdh): has skinny extents
[ 124.250554] BTRFS error (device sdh): open_ctree failed

dmesg entries after running 'mount -a' in emergency mode:
[ 178.317339] BTRFS info (device sdh): force zstd compression, level 2
[ 178.317342] BTRFS info (device sdh): using free space tree
[ 178.317343] BTRFS info (device sdh): has skinny extents

uname -a:
Linux HOSTNAME 5.10.0-2-amd64 #1 SMP Debian 5.10.9-1 (2021-01-20) x86-64 GNU/Linux

btrfs --version:
btrfs-progs v5.10

btrfs fi show /mountpoint:
Label: 'DATA' uuid: {snip}
Total devices 14 FS bytes used 41.94TiB
devid 1 size 2.73TiB used 2.46TiB path /dev/sdh
devid 2 size 7.28TiB used 6.87TiB path /dev/sdm
devid 3 size 2.73TiB used 2.46TiB path /dev/sdk
devid 4 size 9.10TiB used 8.57TiB path /dev/sdj
devid 5 size 9.10TiB used 8.57TiB path /dev/sde
devid 6 size 9.10TiB used 8.57TiB path /dev/sdn
devid 7 size 7.28TiB used 4.65TiB path /dev/sdc
devid 9 size 9.10TiB used 8.57TiB path /dev/sdf
devid 10 size 2.73TiB used 2.21TiB path /dev/sdl
devid 12 size 2.73TiB used 2.20TiB path /dev/sdg
devid 13 size 9.10TiB used 8.57TiB path /dev/sdd
devid 15 size 7.28TiB used 6.75TiB path /dev/sda
devid 16 size 7.28TiB used 6.75TiB path /dev/sdi
devid 17 size 7.28TiB used 6.75TiB path /dev/sdb

With such a large array, the extent tree is considerably large.

And that's causing the mount time problem, as at mount we need to load
each block group item into memory.
When extent tree goes large, the read is mostly random read which is
never a good thing for HDD.

I was pushing skinny block group tree for btrfs, which arrange block
group items into a very compact tree, just like chunk tree.

This should greatly improve the mount performance, but there are several
problems:
- The feature is not yet merged
- The feature needs to convert existing fs to the new tree
For your fs, it may take quite some time

So unfortunately, no good short term solution yet.

THanks,
Qu

Thanks for the information, that's more or less what I was wondering, but didn't really know.

Luckily the solution proposed by Graham appears to be working, and 'solved' the problem for me, allowing my system to boot reliably.

The only remaining issue is the annoyance of boot times (mount times) being so long, but luckily that's not a very big deal for my situation, and I don't need to reboot (mount) very frequently.


Thanks,
--Joshua Villwock

quoted

btrfs fi usage /mountpoint:
Overall:
Device size: 92.78TiB
Device allocated: 83.96TiB
Device unallocated: 8.83TiB
Device missing: 0.00B
Used: 83.94TiB
Free (estimated): 4.42TiB (min: 2.95TiB)
Free (statfs, df): 3.31TiB
Data ratio: 2.00
Metadata ratio: 3.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no

Data,RAID1: Size:41.88TiB, Used:41.877TiB (99.99%)
{snip}

Metadata,RAID1C3: Size:68GiB, Used:63.79GiB (93.81%)
{snip}

System,RAID1C3: Size:32MiB, Used:6.69MiB (20.90%)
{snip}

Unallocated:
{snip}