Thread (3 messages) 3 messages, 2 authors, 2021-06-18

Re: Kernel GPF on RAID6 replace

From: Thiago Ramon <hidden>
Date: 2021-06-18 19:50:44

Setup is a bit unusual, BTRFS is running over BCACHE, whole
unpartitioned disks, here's the raw data, removed the unrelated disks:

lsblk -o NAME,FSTYPE,SIZE,FSUSE%,MOUNTPOINT,UUID,MIN-IO,SCHED,DISC-GRAN,MODEL:

sda                        bcache        7.3T
bc7c0748-4cb4-43cc-ad5d-d736e5e0db82     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache1                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdb                        bcache        7.3T
0a642b3a-78c8-435f-89bd-4fce86c270a7     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache8                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdc                        bcache        7.3T
5f2a88df-1465-464a-96f7-53432f5d4acc     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache9                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdd                        bcache        7.3T
f48df2c8-b665-485b-ae82-ac0cb256c1de     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache11                 btrfs         7.3T    66% /data
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdg                        bcache        7.3T
ba8101e5-6d9e-44a1-8339-3bbbfa9db030     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache6                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdh                        bcache        7.3T
301fb860-2eb3-488c-94b7-7013f2a9ad0c     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache3                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdi                        bcache        7.3T
7ba65110-6a17-4c67-be63-060d2f721a87     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache5                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdj                        bcache        7.3T
36d72ae0-b03d-458c-b75d-c62804ddb240     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache4                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdk                        bcache        7.3T
2b494639-5863-4bb9-b462-686f62e75091     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache2                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdl                        bcache        7.3T
03b62951-3532-40f5-9a67-1ef287caa890     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache0                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdm                        bcache        7.3T
8b978d80-c6c3-426b-a260-86221212610c     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache7                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdn                        bcache        7.3T
5e36e48a-c550-402e-b249-11fae77cc740     4096 mq-deadline        0B
ST8000DM004-2CX1
└─bcache10                 btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
sdp                        bcache      223.6G
517b4831-9c41-4c86-8c8f-f5e5a2d74d27      512 mq-deadline      512B
KINGSTON_SA400S3
├─bcache0                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache1                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache2                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache3                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache4                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache5                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache6                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache7                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache8                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache9                  btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache10                 btrfs         7.3T
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
├─bcache11                 btrfs         7.3T    66% /data
224936e2-df99-4952-952e-9fc8cf87fc34      512                  512B
└─bcache12                 ext4          3.7T    42% /scratch
3590506e-c0f0-45c1-8fd7-3fafe3a73024      512                  512B

btrfs filesystem usage -T /data
WARNING: RAID56 detected, not implemented
Overall:
    Device size:                  94.61TiB
    Device allocated:            488.12GiB
    Device unallocated:           94.13TiB
    Device missing:                7.28TiB
    Used:                        447.63GiB
    Free (estimated):                0.00B      (min: 8.00EiB)
    Data ratio:                       0.00
    Metadata ratio:                   4.00
    Global reserve:              512.00MiB      (used: 0.00B)

                 Data     Metadata  System
Id Path          RAID6    RAID1C4   RAID1C4  Unallocated
-- ------------- -------- --------- -------- -----------
 0 /dev/bcache11        -         -        -     7.28TiB
 1 /dev/bcache2   5.71TiB  41.00GiB        -     1.53TiB
 4 /dev/bcache8   5.71TiB  40.00GiB 32.00MiB     1.53TiB
 5 /dev/bcache0   5.71TiB  41.00GiB        -     1.53TiB
 6 missing        5.71TiB  41.00GiB        -    -5.75TiB
 7 /dev/bcache1   5.71TiB  40.00GiB 32.00MiB     1.53TiB
 8 /dev/bcache9   5.71TiB  41.00GiB        -     1.53TiB
 9 /dev/bcache7   5.71TiB  41.00GiB        -     1.53TiB
10 /dev/bcache10  5.71TiB  40.00GiB 32.00MiB     1.53TiB
11 /dev/bcache6   5.71TiB  41.00GiB        -     1.53TiB
12 /dev/bcache3   5.71TiB  40.00GiB 32.00MiB     1.53TiB
13 /dev/bcache5   5.71TiB  41.00GiB        -     1.53TiB
14 /dev/bcache4   5.71TiB  41.00GiB        -     1.53TiB
-- ------------- -------- --------- -------- -----------
   Total         57.11TiB 122.00GiB 32.00MiB    18.32TiB
   Used          56.95TiB 111.91GiB  2.69MiB

Currently mounted degraded,readonly. Didn't explore much yet, but
overall seems "fine" (as much as fine as a RAID6 missing a disk can
be).

On Fri, Jun 18, 2021 at 4:24 PM Chris Murphy [off-list ref] wrote:
It'd be useful to see the storage stack:

lsblk -o NAME,FSTYPE,SIZE,FSUSE%,MOUNTPOINT,UUID,MIN-IO,SCHED,DISC-GRAN,MODEL

It's fine to omit the UUID. Mainly I just want to see all the physical
devices, their partitions, and how everything is being assembled to
make up this Btrfs file system.

Also useful info

sudo btrfs filesystem usage -T /mntpoint


Chris Murphy
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help