On Mon, Dec 17, 2012 at 04:51:33PM +0100, Sebastien Luttringer wrote:

Hello,

I'm testing Btrfs RAID1 feature on 3 disks of ~10GB. Last one is not
exactly 10GB (would be too easy).

About the test machine, it's a kvm vm running an up-to-date archlinux
with linux 3.7 and btrfs-progs 0.19.20121005.

#uname -a
Linux seblu-btrfs-1 3.7.0-1-ARCH #1 SMP PREEMPT Tue Dec 11 15:05:50 CET
2012 x86_64 GNU/Linux

Filesystem was created with :

# mkfs.btrfs -L test -d raid1 -m raid1 /dev/vd[bcd]

I downloaded a lot of linux kernel tarball and untared in into this
filesystem until it tell me enough:

drwxr-xr-x 1 root root  330 2007-10-09 20:31 linux-2.6.23
-rw-r--r-- 1 root root  44M 2007-10-09 20:48 linux-2.6.23.tar.bz2
drwxr-xr-x 1 root root  344 2008-01-24 22:58 linux-2.6.24
-rw-r--r-- 1 root root  45M 2008-01-24 23:16 linux-2.6.24.tar.bz2
drwxr-xr-x 1 root root  352 2008-04-17 02:49 linux-2.6.25

Some output of btrfs tools

# btrfs fi sh
Label: 'test'  uuid: 7d72c625-4dd7-4db0-b4a2-075e26572b99
	Total devices 3 FS bytes used 11.57GB
	devid    3 size 10.00GB used 9.76GB path /dev/vdd
	devid    2 size 10.00GB used 10.00GB path /dev/vdc
	devid    1 size 9.77GB used 9.77GB path /dev/vdb

# btrfs fi df .
Data, RAID1: total=13.50GB, used=10.56GB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=1.25GB, used=1.01GB
Metadata: total=8.00MB, used=0.00

I have choosen a _raid1_ filesystem, so I expect every disk be able to
die and fs still usuable without data lost.

   This is where your expectations and reality have their differences.

I used chksfv[1] to make checksum of tarball on disk and check my data
are safe.

I killed the first disk via libvirt.
# virsh detach-disk  seblu-btrfs-1 vdb

Btrfs detect missing disk \o/

# btrfs fi sh
Label: 'test'  uuid: 7d72c625-4dd7-4db0-b4a2-075e26572b99
	Total devices 3 FS bytes used 11.57GB
	devid    3 size 10.00GB used 8.01GB path /dev/vdd
	devid    2 size 10.00GB used 9.26GB path /dev/vdc
	*** Some devices missing

Listing all files and cksum on tarballs are good.

# cksfv -f ~/checksum
Everything OK

We have raid1, so we can expect kill one disk (the smaller). Let's go.

# virsh detach-disk  seblu-btrfs-1 vdd

# btrfs fi sh
Label: 'test'  uuid: 7d72c625-4dd7-4db0-b4a2-075e26572b99
	Total devices 3 FS bytes used 11.57GB
	devid    3 size 10.00GB used 8.01GB path /dev/vdd
	devid    2 size 10.00GB used 9.26GB path /dev/vdc
	*** Some devices missing

[snip]

   Yes, this is the expected behaviour.

I get the feeling that RAID1 only allow one disk removing. Which is more
a RAID5 feature.

   The RAID-1 support in btrfs makes exactly two copies of each item
of data, so you can lose at most one disk from the array safely. Lose
any more, and you're likely to have lost data, as you've found out.

I'm afraid Btrfs raid1 will not be working before the end of the world.

   It does work (as you demonstrated with the first disk being
removed) -- but just not as you thought it should. Now, you can argue
that "RAID-1" isn't a good name to use here, but there's no good name
in RAID terminology to describe what we actually have here.

   I believe that Chris has done some work on allowing (tunable)
multiple copies of data in RAID-1 mode, but that's not been published
yet. It was part of the extended RAID work that he was working on. I
think we're hoping for RAID-5 and RAID-6 to arrive in this merge
window, but that's been said before (and it hasn't been stable enough
for release before). I think Chris was going to leave the n-copies
RAID-1 feature for a later release, and just get RAID-5/6 out first.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
      --- "Le Corbusier's plan for improving Paris involved the ---      
                assassination of the city, and its rebirth                
                            as tower blocks."