Re: btrfs device delete /dev/sdc1 /mnt/raid1 user experience

On Mon, Jun 6, 2016 at 10:02 PM, Kai Hendry [off-list ref] wrote:

Sorry I unsubscribed from linux-btrfs@vger.kernel.org since the traffic
was a bit too high for me.

On Tue, 7 Jun 2016, at 11:42 AM, Chris Murphy wrote:

quoted

Your command turned this from a 3 drive volume into a 2 drive volume,
it removed the drive you asked to be removed.

I actually had 2 drives to begin with, but it wouldn't allow me to
delete the drive without adding a new one, so I had 3 drives.

Two drive raid1 is the minimum number of drives, that's why it won't
allow you to delete one.

I think I've confused you by my talk of "RAID-1 over 3 drives". I am not
sure why RAIDX terminology is so confusing. All I want is for the data
on one drive to be mirrored across them all. So if I add X drives, I
want to X exact copies. But IIUC this changes the RAID level and
everyone gets confused what I am talking about.

Yeah you're confused about both RAID in general and how Btrfs raid
works. There is no guarantee of X drives meaning X copies with any
RAID-1. The mkfs.btrfs man page makes it very clear that there are
only ever two copies of data with Btrfs raid1 no matter how many
drives you include.

quoted

You're definitely missing something but once you understand how it
actually works, it'll be interesting if you have suggestions on how
the existing documentation confused you into making this mistake.

Summary: I issued the remove command when I should have just removed it
physically IIUC.

Yes, but that only works with a two drive Btrfs raid1. The minimum
number of drives for successful degraded mount is n-1 drives. You can
only lose 1 drive with Btrfs raid1 volumes, you must have all of the
other drives for it to be mountable degraded.

If I was to do this again, I would unmount the raid1 mount. Take a disk
physically out and then add the new one. So yes, it would be degraded
but then as you mention, the newly drive will sort it out. One would
hope.

Btrfs isn't mature enough to hope for the best. You definitely need to
understand it. There's a lot less you need to understand than mdadm or
LVM plus other file system stuff, but there's still minimum knowledge
required and also there are gotchas that aren't always obvious.

-- 
Chris Murphy