Hello,
I've searched high and low about my issue, but have been unable to
turn up anything like what I'm seeing right now.
A little background: I started using BTRFS over a year ago, in RAID 1
with mixed size drives. A few months ago, I started replacing the
disks with 4 TB drives, and eventually switched over to RAID 6. I am
currently running a 6x4TB RAID6 drive configuration, which should give
me ~14.5 TB
usable, but I'm only getting around 11.
The weird thing is that It seems to completely fill 4/6 of the disks,
while leaving lots of space free on 2 of the disks. I've tried full
filesystem balances, yet the problem continues.
# btrfs fi show
Label: none uuid: 78733087-d597-4301-8efa-8e1df800b108
Total devices 6 FS bytes used 11.59TiB
devid 1 size 3.64TiB used 3.64TiB path /dev/sdd
devid 2 size 3.64TiB used 3.64TiB path /dev/sdg
devid 3 size 3.64TiB used 3.64TiB path /dev/sdf
devid 5 size 3.64TiB used 2.92TiB path /dev/sda
devid 6 size 3.64TiB used 1.48TiB path /dev/sdb
devid 7 size 3.64TiB used 3.64TiB path /dev/sdc
btrfs-progs v4.2.3
# btrfs fi df /mnt/data
Data, RAID6: total=11.67TiB, used=11.58TiB
System, RAID6: total=64.00MiB, used=1.70MiB
Metadata, RAID6: total=15.58GiB, used=13.89GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
dan@Morpheus:/mnt/data/temp$ sudo btrfs fi usage /mnt/data
# btrfs fi usage /mnt/data
WARNING: RAID56 detected, not implemented
WARNING: RAID56 detected, not implemented
WARNING: RAID56 detected, not implemented
Overall:
Device size: 21.83TiB
Device allocated: 0.00B
Device unallocated: 21.83TiB
Device missing: 0.00B
Used: 0.00B
Free (estimated): 0.00B (min: 8.00EiB)
Data ratio: 0.00
Metadata ratio: 0.00
Global reserve: 512.00MiB (used: 0.00B)
Data,RAID6: Size:11.67TiB, Used:11.58TiB
/dev/sda 2.92TiB
/dev/sdb 1.48TiB
/dev/sdc 3.63TiB
/dev/sdd 3.63TiB
/dev/sdf 3.63TiB
/dev/sdg 3.63TiB
Metadata,RAID6: Size:15.58GiB, Used:13.89GiB
/dev/sda 4.05GiB
/dev/sdb 1.50GiB
/dev/sdc 5.01GiB
/dev/sdd 5.01GiB
/dev/sdf 5.01GiB
/dev/sdg 5.01GiB
System,RAID6: Size:64.00MiB, Used:1.70MiB
/dev/sda 16.00MiB
/dev/sdb 16.00MiB
/dev/sdc 16.00MiB
/dev/sdd 16.00MiB
/dev/sdf 16.00MiB
/dev/sdg 16.00MiB
Unallocated:
/dev/sda 733.65GiB
/dev/sdb 2.15TiB
/dev/sdc 1.02MiB
/dev/sdd 1.02MiB
/dev/sdf 1.02MiB
/dev/sdg 1.02MiB
Can anyone shed some light on why a full balance (sudo btrfs balance
start /mnt/data) doesnt seem to straighten this out? Any and all help
is appreciated.
Thanks!
Dan Blazejewski posted on Tue, 16 Feb 2016 15:20:12 -0500 as excerpted:
A little background: I started using BTRFS over a year ago, in RAID 1
with mixed size drives. A few months ago, I started replacing the disks
with 4 TB drives, and eventually switched over to RAID 6. I am currently
running a 6x4TB RAID6 drive configuration, which should give me ~14.5 TB
usable, but I'm only getting around 11.
The weird thing is that It seems to completely fill 4/6 of the disks,
while leaving lots of space free on 2 of the disks. I've tried full
filesystem balances, yet the problem continues.
# btrfs fi show
Label: none uuid: 78733087-d597-4301-8efa-8e1df800b108
Total devices 6 FS bytes used 11.59TiB
devid 1 size 3.64TiB used 3.64TiB path /dev/sdd
devid 2 size 3.64TiB used 3.64TiB path /dev/sdg
devid 3 size 3.64TiB used 3.64TiB path /dev/sdf
devid 5 size 3.64TiB used 2.92TiB path /dev/sda
devid 6 size 3.64TiB used 1.48TiB path /dev/sdb
devid 7 size 3.64TiB used 3.64TiB path /dev/sdc
btrfs-progs v4.2.3
# btrfs fi df /mnt/data
Data, RAID6: total=11.67TiB, used=11.58TiB
System, RAID6: total=64.00MiB, used=1.70MiB
Metadata, RAID6: total=15.58GiB, used=13.89GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi usage /mnt/data
WARNING: RAID56 detected, not implemented
Your btrfs-progs is old and I don't see any indication of kernel version
at all, but I'll guess it's old as well. Particularly for raid56 mode,
which still isn't to the maturity level of the rest of btrfs, using
current kernel and btrfs-progs is *very* strongly recommended.
Among other things, current userspace 4.4 btrfs fi usage should support
raid56 mode properly, now. Also, with newer userspace and kernel, btrfs
balance supports the stripes= filter, which appears to be what you're
looking for, to rebalance to full-width stripes anything that's not yet
full width, thereby evening out your usage.
A full balance /should/ do it as well, I believe, but with raid56 support
still not yet at the maturity level of btrfs in general, it's likely your
version is old and buggy in that regard.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Dan Blazejewski wrote on 2016/02/16 15:20 -0500:
Hello,
I've searched high and low about my issue, but have been unable to
turn up anything like what I'm seeing right now.
A little background: I started using BTRFS over a year ago, in RAID 1
with mixed size drives. A few months ago, I started replacing the
disks with 4 TB drives, and eventually switched over to RAID 6. I am
currently running a 6x4TB RAID6 drive configuration, which should give
me ~14.5 TB
usable, but I'm only getting around 11.
The weird thing is that It seems to completely fill 4/6 of the disks,
while leaving lots of space free on 2 of the disks. I've tried full
filesystem balances, yet the problem continues.
# btrfs fi show
Label: none uuid: 78733087-d597-4301-8efa-8e1df800b108
Total devices 6 FS bytes used 11.59TiB
devid 1 size 3.64TiB used 3.64TiB path /dev/sdd
devid 2 size 3.64TiB used 3.64TiB path /dev/sdg
devid 3 size 3.64TiB used 3.64TiB path /dev/sdf
devid 5 size 3.64TiB used 2.92TiB path /dev/sda
devid 6 size 3.64TiB used 1.48TiB path /dev/sdb
devid 7 size 3.64TiB used 3.64TiB path /dev/sdc
btrfs-progs v4.2.3
Your space really used up, as it can't found *at least 4* disk with enough space to allocate a new chunk.
As 4 devices in your array is already filled, the rest 2 is of no means for RAID6, and can only be allocated with Single/RAID1/RAID0.
But the real problem is, why your devices get such a unbalanced layout.
Normally, for RAID5/6, it will allocate chunks using all disk with available space, and since all your devices are in the same size, it should result very balanced allocation.
How did you convert to current RAID6? Did it involves balance from already some used disks?
Thanks,
Qu
# btrfs fi df /mnt/data
Data, RAID6: total=11.67TiB, used=11.58TiB
System, RAID6: total=64.00MiB, used=1.70MiB
Metadata, RAID6: total=15.58GiB, used=13.89GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
dan@Morpheus:/mnt/data/temp$ sudo btrfs fi usage /mnt/data
# btrfs fi usage /mnt/data
WARNING: RAID56 detected, not implemented
WARNING: RAID56 detected, not implemented
WARNING: RAID56 detected, not implemented
Overall:
Device size: 21.83TiB
Device allocated: 0.00B
Device unallocated: 21.83TiB
Device missing: 0.00B
Used: 0.00B
Free (estimated): 0.00B (min: 8.00EiB)
Data ratio: 0.00
Metadata ratio: 0.00
Global reserve: 512.00MiB (used: 0.00B)
Data,RAID6: Size:11.67TiB, Used:11.58TiB
/dev/sda 2.92TiB
/dev/sdb 1.48TiB
/dev/sdc 3.63TiB
/dev/sdd 3.63TiB
/dev/sdf 3.63TiB
/dev/sdg 3.63TiB
Metadata,RAID6: Size:15.58GiB, Used:13.89GiB
/dev/sda 4.05GiB
/dev/sdb 1.50GiB
/dev/sdc 5.01GiB
/dev/sdd 5.01GiB
/dev/sdf 5.01GiB
/dev/sdg 5.01GiB
System,RAID6: Size:64.00MiB, Used:1.70MiB
/dev/sda 16.00MiB
/dev/sdb 16.00MiB
/dev/sdc 16.00MiB
/dev/sdd 16.00MiB
/dev/sdf 16.00MiB
/dev/sdg 16.00MiB
Unallocated:
/dev/sda 733.65GiB
/dev/sdb 2.15TiB
/dev/sdc 1.02MiB
/dev/sdd 1.02MiB
/dev/sdf 1.02MiB
/dev/sdg 1.02MiB
Can anyone shed some light on why a full balance (sudo btrfs balance
start /mnt/data) doesnt seem to straighten this out? Any and all help
is appreciated.
Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html