Raid 1, can't get the second disk added back in.

Sometime in the distant past, I lost a member of my raid 1 group.  Here's some output:

root@chinaberry:~# mdadm /dev/md0 --add /dev/hdb5
mdadm: Cannot open /dev/hdb5: Device or resource busy
root@chinaberry:~# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 hda5[1]
     93771264 blocks [2/1] [_U]

unused devices: <none>
root@chinaberry:~# cat /etc/mdadm.conf
DEVICE /dev/hdb5 /dev/hda5
ARRAY /dev/md0 level=raid1 num-devices=2

root@chinaberry:~# fdisk -l /dev/hdb

Disk /dev/hdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hdb1               1         973     7815591   83  Linux
/dev/hdb2             974        1946     7815622+  83  Linux
/dev/hdb3            1947        2919     7815622+  83  Linux
/dev/hdb4            2920       14593    93771405    5  Extended
/dev/hdb5            2920       14593    93771373+  83  Linux
root@chinaberry:~# fdisk -l /dev/hda

Disk /dev/hda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1         973     7815591   83  Linux
/dev/hda2             974        1946     7815622+  83  Linux
/dev/hda3            1947        2919     7815622+  83  Linux
/dev/hda4            2920       14593    93771405    5  Extended
/dev/hda5            2920       14593    93771373+  83  Linux

I attempted to boot single user mode and fsck /dev/hdb5 but single user doesn't seem to do what it used to, for some reason /dev/md0 was mounted.  However I did boot Knoppix CD and do a fsck -f /dev/hdb5.  It had no errors and when I rebooted back it still wouldn't install.

73 more lines

----- Message from jim@fayettedigital.com ---------
    Date: Sat, 05 Jan 2008 11:05:00 -0500
    From: Jim [off-list ref]
Reply-To: jim@fayettedigital.com
 Subject: Raid 1, can't get the second disk added back in.
      To: linux-raid@vger.kernel.org

Sometime in the distant past, I lost a member of my raid 1 group.
Here's some output:

root@chinaberry:~# mdadm /dev/md0 --add /dev/hdb5
mdadm: Cannot open /dev/hdb5: Device or resource busy
root@chinaberry:~# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 hda5[1]
     93771264 blocks [2/1] [_U]

unused devices: <none>
root@chinaberry:~# cat /etc/mdadm.conf
DEVICE /dev/hdb5 /dev/hda5
ARRAY /dev/md0 level=raid1 num-devices=2

root@chinaberry:~# fdisk -l /dev/hdb

Disk /dev/hdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hdb1               1         973     7815591   83  Linux
/dev/hdb2             974        1946     7815622+  83  Linux
/dev/hdb3            1947        2919     7815622+  83  Linux
/dev/hdb4            2920       14593    93771405    5  Extended
/dev/hdb5            2920       14593    93771373+  83  Linux
root@chinaberry:~# fdisk -l /dev/hda

Disk /dev/hda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1         973     7815591   83  Linux
/dev/hda2             974        1946     7815622+  83  Linux
/dev/hda3            1947        2919     7815622+  83  Linux
/dev/hda4            2920       14593    93771405    5  Extended
/dev/hda5            2920       14593    93771373+  83  Linux

59 more quoted lines

----- End message from jim@fayettedigital.com -----

Since /dev/hdb5 has been part of this array before you should use --re-add instead of --add.
Kind regards,
Alex.

========================================================================
#    _  __          _ __     http://www.nagilum.org/ \n icq://69646724 #
#   / |/ /__ ____ _(_) /_ ____ _  nagilum@nagilum.org \n +491776461165 #
#  /    / _ `/ _ `/ / / // /  ' \  Amiga (68k/PPC): AOS/NetBSD/Linux   #
# /_/|_/\_,_/\_, /_/_/\_,_/_/_/_/   Mac (PPC): MacOS-X / NetBSD /Linux #
#           /___/     x86: FreeBSD/Linux/Solaris/Win2k  ARM9: EPOC EV6 #
========================================================================


----------------------------------------------------------------
cakebox.homeunix.net - all the machine one needs..

Nagilum wrote:

Since /dev/hdb5 has been part of this array before you should use --re-add instead of --add.
Kind regards,
Alex.

OK

root@chinaberry:~# mdadm  /dev/md0 --re-add /dev/hdb5
mdadm: Cannot open /dev/hdb5: Device or resource busy

Jim

----- Message from jim@fayettedigital.com ---------
    Date: Sun, 06 Jan 2008 06:13:06 -0500
    From: Jim [off-list ref]

root@chinaberry:~# mdadm  /dev/md0 --re-add /dev/hdb5
mdadm: Cannot open /dev/hdb5: Device or resource busy

Have you tried stopping (-S) /dev/md0 and then re-assemble the array (-A)?


========================================================================
#    _  __          _ __     http://www.nagilum.org/ \n icq://69646724 #
#   / |/ /__ ____ _(_) /_ ____ _  nagilum@nagilum.org \n +491776461165 #
#  /    / _ `/ _ `/ / / // /  ' \  Amiga (68k/PPC): AOS/NetBSD/Linux   #
# /_/|_/\_,_/\_, /_/_/\_,_/_/_/_/   Mac (PPC): MacOS-X / NetBSD /Linux #
#           /___/     x86: FreeBSD/Linux/Solaris/Win2k  ARM9: EPOC EV6 #
========================================================================


----------------------------------------------------------------
cakebox.homeunix.net - all the machine one needs..

Nagilum wrote:

Yes, I did that yesterday.  Something is holding on to /dev/hdb5, 'cause I got the device busy message from that attempt too.  I think I'm going to remove the partition and see what happens.

Thanks,
Jim.