Error in rebuild of two "layered" md devices in container

Hi Neil,

found another bug.

- Created a container with six disks
- Created two md devices in it:

mdadm -CR /dev/md0 -l 6 -n 6 -z 50M
mdadm -CR /dev/md1 -l 5 -n 6 -z 50M

The md devices are "layered" in the container across all disks.

They both get build and are online.

- Fail one disk, both md devices are affected
- Remove disk
- Clear superblock of removed disk
- Add disk again (in essence, I just added a spare disk)

Now comes the error:

- md0 is rebuild
- md1 is NOT rebuild

Personalities : [raid6] [raid5] [raid4] [raid1]
md1 : active raid6 sde[6] sdg[5] sdf[4] sdd[2] sdc[1] sdb[0]
      204800 blocks super external:/md127/1 level 6, 512k chunk,
algorithm 10 [6/6] [UUUUUU]

md0 : active raid5 sdb[6] sdg[5] sdf[4] sdd[2] sdc[1]
      256000 blocks super external:/md127/0 level 5, 512k chunk,
algorithm 2 [6/5] [UUU_UU]

md127 : inactive sde[3](S) sdb[0](S) sdg[5](S) sdf[4](S) sdd[2](S) sdc[1](S)
      196608 blocks super external:ddf

unused devices: <none>

 Physical Disks : 6
      Number    RefNo      Size       Device      Type/State
         0    cb5ea6c1   1015808K /dev/sdc        active/Online
         1    a7f8ed2f   1015808K /dev/sdd        active/Online
         2    f769a815   1015808K /dev/sdf        active/Online
         3    025e6835   1015808K /dev/sdg        active/Online
         4    b22e9e4d   1015808K /dev/sdb        active/Online
         5    b4cccecc   1015808K /dev/sde        active/Online


Albert

Extra note:

as the mdadm -f command has to be addressed to a subarray, only this
one is going to be rebuild, even when other subarrays are depending on
this disk, as is in this case.

In the case below I failed the disk using mdadm -f /dev/md0 /dev/....

Albert

On 1 August 2012 19:52, Albert Pauw [off-list ref] wrote:

Hi Neil,

found another bug.

- Created a container with six disks
- Created two md devices in it:

mdadm -CR /dev/md0 -l 6 -n 6 -z 50M
mdadm -CR /dev/md1 -l 5 -n 6 -z 50M

The md devices are "layered" in the container across all disks.

They both get build and are online.

- Fail one disk, both md devices are affected
- Remove disk
- Clear superblock of removed disk
- Add disk again (in essence, I just added a spare disk)

Now comes the error:

- md0 is rebuild
- md1 is NOT rebuild

Personalities : [raid6] [raid5] [raid4] [raid1]
md1 : active raid6 sde[6] sdg[5] sdf[4] sdd[2] sdc[1] sdb[0]
      204800 blocks super external:/md127/1 level 6, 512k chunk,
algorithm 10 [6/6] [UUUUUU]

md0 : active raid5 sdb[6] sdg[5] sdf[4] sdd[2] sdc[1]
      256000 blocks super external:/md127/0 level 5, 512k chunk,
algorithm 2 [6/5] [UUU_UU]

md127 : inactive sde[3](S) sdb[0](S) sdg[5](S) sdf[4](S) sdd[2](S) sdc[1](S)
      196608 blocks super external:ddf

unused devices: <none>

 Physical Disks : 6
      Number    RefNo      Size       Device      Type/State
         0    cb5ea6c1   1015808K /dev/sdc        active/Online
         1    a7f8ed2f   1015808K /dev/sdd        active/Online
         2    f769a815   1015808K /dev/sdf        active/Online
         3    025e6835   1015808K /dev/sdg        active/Online
         4    b22e9e4d   1015808K /dev/sdb        active/Online
         5    b4cccecc   1015808K /dev/sde        active/Online


Albert

On Wed, 1 Aug 2012 19:52:51 +0200 Albert Pauw [off-list ref] wrote:

Hi Neil,

found another bug.

- Created a container with six disks
- Created two md devices in it:

mdadm -CR /dev/md0 -l 6 -n 6 -z 50M
mdadm -CR /dev/md1 -l 5 -n 6 -z 50M

The md devices are "layered" in the container across all disks.

They both get build and are online.

- Fail one disk, both md devices are affected
- Remove disk
- Clear superblock of removed disk
- Add disk again (in essence, I just added a spare disk)

Now comes the error:

- md0 is rebuild
- md1 is NOT rebuild

The reason for this is somewhat messy.
mdadm will currently only add a 'spare' device to an array which needs a
replacement device.
In DDF the whole device is either 'active' or 'spare'.  There isn't a concept
of 'partly active, partly spare'.
So when mdadm adds part of the disk to one array it stops being spare and
started being active.  So when mdadm looks for a spare to add to the second
array, there are no spare devices.

I can hack around it by allowing any non-failed device to be considered as a
spare but I need to find a better solution.  That might take a while.  I've
made a note on my to-do list, but it is a rather long list.

Thanks,
NeilBrown

diff --git a/super-ddf.c b/super-ddf.c
index d006a04..11b98f7 100644
--- a/super-ddf.c
+++ b/super-ddf.c

@@ -2616,7 +2616,7 @@ static int validate_geometry_ddf(struct supertype *st,
 	if (chunk && *chunk == UnSet)
 		*chunk = DEFAULT_CHUNK;
 
-
+	if (level == -1000000) level = LEVEL_CONTAINER;
 	if (level == LEVEL_CONTAINER) {
 		/* Must be a fresh device to add to a container */
 		return validate_geometry_ddf_container(st, level, layout,

@@ -3701,6 +3701,10 @@ static struct mdinfo *ddf_activate_spare(struct active_array *a,
 			} else if (ddf->phys->entries[dl->pdnum].type &
 				   __cpu_to_be16(DDF_Global_Spare)) {
 				is_global = 1;
+			} else if (!(ddf->phys->entries[dl->pdnum].state &
+				     __cpu_to_be16(DDF_Failed))) {
+				/* we can possibly use some of this */
+				is_global = 1;
 			}
 			if ( ! (is_dedicated ||
 				(is_global && global_ok))) {

Hi Neil,

as a first test I can confirm that this fixes the problem with the layered md devices in a container.

So far so good on this.

Thanks,

regards,

Albert

On 08/15/2012 01:43 AM, NeilBrown wrote: