mdadm -add doesn't start rebuilding array

2 messages, 2 authors, 2012-08-20 · open the first message on its own page

mdadm -add doesn't start rebuilding array

From: Sergiusz Brzeziński <hidden>
Date: 2012-08-20 09:36:10 

Hi,

My system is Ubuntu 12.04, kernel 3.2.0-27. mdadm: 3.2.3

If I do:

# mdadm /dev/md0 -a /dev/sdc3

then nothing happen! No message on command line, no info in logs!

Man mdadm says:

-a, --add
hot-add listed devices.  If a device appears to have recently been part of the array (possibly it failed or was removed) the device is  re-added  as  describe  in  the  next point.   If  that  fails  or the device was never part of the array, the device is added as a hot-spare.  If the array is degraded, it will immediately start to rebuild data onto that spare.

But array doesn't wont to rebuild.

But not exactly. The bad info is, that sometimes it works and sometimes it doesn't.

And if I restart the system, the array SOMETIMES start rebuilding and somethimes doesn't!

There is no information what's up (comnand line or logs) so I don't know what to do.



I describe bellow the whole procedure:

1.
There ist a working Raid1 array:

# cat /proc/mdstat

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sda3[2] sdb3[3]
      115999672 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

# mdadm --detail /dev/md0

/dev/md0:
        Version : 1.0
  Creation Time : Wed Aug  1 15:45:56 2012
     Raid Level : raid1
     Array Size : 115999672 (110.63 GiB 118.78 GB)
  Used Dev Size : 115999672 (110.63 GiB 118.78 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Aug 20 11:10:00 2012
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : linux:0
           UUID : b7407176:2e88c73d:2c85e940:05ff7e06
         Events : 5897

    Number   Major   Minor   RaidDevice State
       3       8       19        0      active sync   /dev/sdb3
       2       8        3        1      active sync   /dev/sda3


2.
I remove (phisicaly) the "/dev/sdb" drive of the box (hot-swap)

# cat /proc/mdstat

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sda3[2] sdb3[3](F)
      115999672 blocks super 1.0 [2/1] [_U]
      bitmap: 1/1 pages [4KB], 65536KB chunk

# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.0
  Creation Time : Wed Aug  1 15:45:56 2012
     Raid Level : raid1
     Array Size : 115999672 (110.63 GiB 118.78 GB)
  Used Dev Size : 115999672 (110.63 GiB 118.78 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Aug 20 11:11:58 2012
          State : active, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0

           Name : linux:0
           UUID : b7407176:2e88c73d:2c85e940:05ff7e06
         Events : 5908

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       2       8        3        1      active sync   /dev/sda3

       3       8       19        -      faulty spare


3.
I insert the drive again - it become "/dev/sdc" instead of "/dev/sdb"

4.
And now I do the following to rebuild the array:

# mdadm /dev/md0 -a /dev/sdc3

mdadm: /dev/sdc3 reports being an active member for /dev/md0, but a --re-add fails.
mdadm: not performing --add as that would convert /dev/sdc3 in to a spare.
mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdc3" first.

I have never seen this messages before (on another systems). But this is not a problem. I do, what they suggest:

# mdadm --zero-superblock /dev/sdc3

# mdadm --examine /dev/sdc3
mdadm: No md superblock detected on /dev/sdc3.

5.
And now I try again rebuild the array:

# mdadm /dev/md0 -a /dev/sdc3

... and nothing happen! No message, no info. I repeat some times the command.

I also try this:

# mdadm /dev/md0 -a -vv --force /dev/sdc3

... also nothing. Somethimes, if I repeat the command one by one, in logs appear the line:

Aug 20 11:30:52 serwer-linmot kernel: [ 1978.111282] md: export_rdev(sdc3)

... and nothing else


# mdadm --examine /dev/sdc3
/dev/sdc3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : b7407176:2e88c73d:2c85e940:05ff7e06
           Name : linux:0
  Creation Time : Wed Aug  1 15:45:56 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 231999344 (110.63 GiB 118.78 GB)
     Array Size : 231999344 (110.63 GiB 118.78 GB)
   Super Offset : 231999472 sectors
          State : clean
    Device UUID : f1a1f291:e10d85cc:b574c6d2:45fc0c5d

Internal Bitmap : -8 sectors from superblock
    Update Time : Mon Aug 20 11:17:07 2012
       Checksum : 71bcac04 - correct
         Events : 0


   Device Role : spare
   Array State : .A ('A' == active, '.' == missing)


# mdadm --detail /dev/md0

/dev/md0:
        Version : 1.0
  Creation Time : Wed Aug  1 15:45:56 2012
     Raid Level : raid1
     Array Size : 115999672 (110.63 GiB 118.78 GB)
  Used Dev Size : 115999672 (110.63 GiB 118.78 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Aug 20 11:18:43 2012
          State : active, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0

           Name : linux:0
           UUID : b7407176:2e88c73d:2c85e940:05ff7e06
         Events : 5946

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       2       8        3        1      active sync   /dev/sda3

       3       8       19        -      faulty spare


Can anyone help?

many thanks in advance

Sergiusz Brzeziński














--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mdadm -add doesn't start rebuilding array

From: NeilBrown <hidden>
Date: 2012-08-20 22:30:40 

On Mon, 20 Aug 2012 11:36:10 +0200 Sergiusz Brzeziński
[off-list ref] wrote:

Hi,

My system is Ubuntu 12.04, kernel 3.2.0-27. mdadm: 3.2.3

If I do:

# mdadm /dev/md0 -a /dev/sdc3

then nothing happen! No message on command line, no info in logs!

Man mdadm says:

-a, --add
hot-add listed devices.  If a device appears to have recently been part of the 
array (possibly it failed or was removed) the device is  re-added  as  describe 
  in  the  next point.   If  that  fails  or the device was never part of the 
array, the device is added as a hot-spare.  If the array is degraded, it will 
immediately start to rebuild data onto that spare.

But array doesn't wont to rebuild.

But not exactly. The bad info is, that sometimes it works and sometimes it doesn't.

And if I restart the system, the array SOMETIMES start rebuilding and somethimes 
doesn't!

There is no information what's up (comnand line or logs) so I don't know what to do.



I describe bellow the whole procedure:

1.
There ist a working Raid1 array:

# cat /proc/mdstat

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] 
[raid10]
md0 : active raid1 sda3[2] sdb3[3]
       115999672 blocks super 1.0 [2/2] [UU]
       bitmap: 0/1 pages [0KB], 65536KB chunk

# mdadm --detail /dev/md0

/dev/md0:
         Version : 1.0
   Creation Time : Wed Aug  1 15:45:56 2012
      Raid Level : raid1
      Array Size : 115999672 (110.63 GiB 118.78 GB)
   Used Dev Size : 115999672 (110.63 GiB 118.78 GB)
    Raid Devices : 2
   Total Devices : 2
     Persistence : Superblock is persistent

   Intent Bitmap : Internal

     Update Time : Mon Aug 20 11:10:00 2012
           State : active
  Active Devices : 2
Working Devices : 2
  Failed Devices : 0
   Spare Devices : 0

            Name : linux:0
            UUID : b7407176:2e88c73d:2c85e940:05ff7e06
          Events : 5897

     Number   Major   Minor   RaidDevice State
        3       8       19        0      active sync   /dev/sdb3
        2       8        3        1      active sync   /dev/sda3


2.
I remove (phisicaly) the "/dev/sdb" drive of the box (hot-swap)

# cat /proc/mdstat

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] 
[raid10]
md0 : active raid1 sda3[2] sdb3[3](F)
       115999672 blocks super 1.0 [2/1] [_U]
       bitmap: 1/1 pages [4KB], 65536KB chunk

# mdadm --detail /dev/md0
/dev/md0:
         Version : 1.0
   Creation Time : Wed Aug  1 15:45:56 2012
      Raid Level : raid1
      Array Size : 115999672 (110.63 GiB 118.78 GB)
   Used Dev Size : 115999672 (110.63 GiB 118.78 GB)
    Raid Devices : 2
   Total Devices : 2
     Persistence : Superblock is persistent

   Intent Bitmap : Internal

     Update Time : Mon Aug 20 11:11:58 2012
           State : active, degraded
  Active Devices : 1
Working Devices : 1
  Failed Devices : 1
   Spare Devices : 0

            Name : linux:0
            UUID : b7407176:2e88c73d:2c85e940:05ff7e06
          Events : 5908

     Number   Major   Minor   RaidDevice State
        0       0        0        0      removed
        2       8        3        1      active sync   /dev/sda3

        3       8       19        -      faulty spare


3.
I insert the drive again - it become "/dev/sdc" instead of "/dev/sdb"

4.
And now I do the following to rebuild the array:

# mdadm /dev/md0 -a /dev/sdc3

mdadm: /dev/sdc3 reports being an active member for /dev/md0, but a --re-add fails.
mdadm: not performing --add as that would convert /dev/sdc3 in to a spare.
mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdc3" first.

I have never seen this messages before (on another systems). But this is not a 
problem. I do, what they suggest:

# mdadm --zero-superblock /dev/sdc3

# mdadm --examine /dev/sdc3
mdadm: No md superblock detected on /dev/sdc3.

5.
And now I try again rebuild the array:

# mdadm /dev/md0 -a /dev/sdc3

... and nothing happen! No message, no info. I repeat some times the command.

I also try this:

# mdadm /dev/md0 -a -vv --force /dev/sdc3

... also nothing. Somethimes, if I repeat the command one by one, in logs appear 
the line:

Aug 20 11:30:52 serwer-linmot kernel: [ 1978.111282] md: export_rdev(sdc3)

... and nothing else


# mdadm --examine /dev/sdc3
/dev/sdc3:
           Magic : a92b4efc
         Version : 1.0
     Feature Map : 0x1
      Array UUID : b7407176:2e88c73d:2c85e940:05ff7e06
            Name : linux:0
   Creation Time : Wed Aug  1 15:45:56 2012
      Raid Level : raid1
    Raid Devices : 2

  Avail Dev Size : 231999344 (110.63 GiB 118.78 GB)
      Array Size : 231999344 (110.63 GiB 118.78 GB)
    Super Offset : 231999472 sectors
           State : clean
     Device UUID : f1a1f291:e10d85cc:b574c6d2:45fc0c5d

Internal Bitmap : -8 sectors from superblock
     Update Time : Mon Aug 20 11:17:07 2012
        Checksum : 71bcac04 - correct
          Events : 0


    Device Role : spare
    Array State : .A ('A' == active, '.' == missing)


# mdadm --detail /dev/md0

/dev/md0:
         Version : 1.0
   Creation Time : Wed Aug  1 15:45:56 2012
      Raid Level : raid1
      Array Size : 115999672 (110.63 GiB 118.78 GB)
   Used Dev Size : 115999672 (110.63 GiB 118.78 GB)
    Raid Devices : 2
   Total Devices : 2
     Persistence : Superblock is persistent

   Intent Bitmap : Internal

     Update Time : Mon Aug 20 11:18:43 2012
           State : active, degraded
  Active Devices : 1
Working Devices : 1
  Failed Devices : 1
   Spare Devices : 0

            Name : linux:0
            UUID : b7407176:2e88c73d:2c85e940:05ff7e06
          Events : 5946

     Number   Major   Minor   RaidDevice State
        0       0        0        0      removed
        2       8        3        1      active sync   /dev/sda3

        3       8       19        -      faulty spare


Can anyone help?
I cannot see why that would happen.
It seems to be failing somewhere in bind_rdev_to_array, but I don't know
where.
Can you run
  strace -o /tmp/strace mdadm /dev/md0 --add /dev/sdc3
and post the output ... or at last the tail end of the output.  There should
be an ioctl which fails and I need to know what the error was (EEXIST,
EINVAL, ENOSPC are the likely ones).  But don't just post that line, include
at least the last 100 lines.

NeilBrown
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help