mdadm: spare rebuilding

Started by Jon Buckingham on Jun 24, 2008

open the first message on its own page

Message details and related content
Hi,

I've rebuilt my server from scratch, and have 4 raided partitions.

All went fine, however one of the partitions (a raid 5) only
appears to be using 3 out of 4 disks.
mdadm indicates that the unused disk is "spare rebuilding",
but after tens of hours and a reboot it's status is unchanged.

There is no significant activity by the relevant process
(md3_raid5) - <1% cpu usage, the same as the other similar processes.

I have tried removing the "spare" disk and re-adding it, but get...
nas:~ # mdadm /dev/md3 --remove /dev/sdd4
mdadm: hot remove failed for /dev/sdd4: Device or resource busy

nas:~ # mdadm /dev/md3 --add /dev/sdd4
mdadm: Cannot open /dev/sdd4: Device or resource busy

There are no obvious errors from the boot log.

Do you have any ideas how to get all 4 disks used, what the
issue might be etc? Or am I just impatient?!

Various logs etc appended.

Thanks

Jon B

----------------------------------
opensuse 10.3

nas:~ # uname -a
Linux nas 2.6.22.5-31-default #1 SMP 2007/09/21 22:29:00 UTC i686 athlon i386 GNU/Linux

nas:~ # rpm -qa | grep mdadm
mdadm-2.6.2-16
----------------------------------
nas:~ # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md3 : active raid5 sda4[0] sdd4[4] sdc4[2] sdb4[1]
      576435840 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/3] [UUU_]
      bitmap: 2/184 pages [8KB], 512KB chunk

md1 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      41945640 blocks super 1.0 [4/4] [UUUU]
      bitmap: 0/161 pages [0KB], 128KB chunk

md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      10490340 blocks super 1.0 [4/4] [UUUU]
      bitmap: 1/161 pages [4KB], 32KB chunk

md2 : active(auto-read-only) raid5 sda3[0] sdd3[4] sdc3[2] sdb3[1]
      1590144 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/9 pages [0KB], 32KB chunk

unused devices: <none>
----------------------------------
nas:~ # mdadm --detail /dev/md3
/dev/md3:
        Version : 01.00.03
  Creation Time : Mon Jun 23 22:03:45 2008
     Raid Level : raid5
     Array Size : 576435840 (549.73 GiB 590.27 GB)
  Used Dev Size : 384290560 (183.24 GiB 196.76 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 3
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Tue Jun 24 21:37:33 2008
          State : active, degraded
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 128K

           Name : 3
           UUID : 17ae4fee:d1380d07:c3265e31:3c77f88f
         Events : 1256

    Number   Major   Minor   RaidDevice State
       0       8        4        0      active sync   /dev/sda4
       1       8       20        1      active sync   /dev/sdb4
       2       8       36        2      active sync   /dev/sdc4
       4       8       52        3      spare rebuilding   /dev/sdd4
----------------------------------
nas:~ # mdadm -E /dev/sdd4
/dev/sdd4:
          Magic : a92b4efc
        Version : 01
    Feature Map : 0x3
     Array UUID : 17ae4fee:d1380d07:c3265e31:3c77f88f
           Name : 3
  Creation Time : Mon Jun 23 22:03:45 2008
     Raid Level : raid5
   Raid Devices : 4

  Used Dev Size : 384290720 (183.24 GiB 196.76 GB)
     Array Size : 1152871680 (549.73 GiB 590.27 GB)
      Used Size : 384290560 (183.24 GiB 196.76 GB)
   Super Offset : 384290848 sectors
Recovery Offset : 48750592 sectors
          State : clean
    Device UUID : 086e1682:4ba0454f:673f8a77:b56f3b92

Internal Bitmap : -93 sectors from superblock
    Update Time : Tue Jun 24 21:47:17 2008
       Checksum : 135f5478 - correct
         Events : 1258

         Layout : left-symmetric
     Chunk Size : 128K

    Array Slot : 4 (0, 1, 2, failed, 3)
   Array State : uuuU 1 failed
----------------------------------
nas:~ # mdadm -E /dev/sda4
/dev/sda4:
          Magic : a92b4efc
        Version : 01
    Feature Map : 0x1
     Array UUID : 17ae4fee:d1380d07:c3265e31:3c77f88f
           Name : 3
  Creation Time : Mon Jun 23 22:03:45 2008
     Raid Level : raid5
   Raid Devices : 4

  Used Dev Size : 384290720 (183.24 GiB 196.76 GB)
     Array Size : 1152871680 (549.73 GiB 590.27 GB)
      Used Size : 384290560 (183.24 GiB 196.76 GB)
   Super Offset : 384290848 sectors
          State : clean
    Device UUID : 6ed4d11c:2092f54b:8d530f91:c1813c49

Internal Bitmap : -93 sectors from superblock
    Update Time : Tue Jun 24 21:47:17 2008
       Checksum : 7868f2de - correct
         Events : 1258

         Layout : left-symmetric
     Chunk Size : 128K

    Array Slot : 0 (0, 1, 2, failed, 3)
   Array State : Uuuu 1 failed
----------------------------------

nas:~ # grep -i -C6 raid /var/log/boot.msg
<6>ata6.00: ATA-7: WDC WD2500YD-01NVB1, 10.02E01, max UDMA/133
<6>ata6.00: 490234752 sectors, multi 16: LBA48 NCQ (depth 0/1)
<6>ata6.00: configured for UDMA/133
<5>scsi 4:0:0:0: Direct-Access     ATA      WDC WD2500YD-01N 10.0 PQ: 0 ANSI: 5
<5>scsi 5:0:0:0: Direct-Access     ATA      WDC WD2500YD-01N 10.0 PQ: 0 ANSI: 5
<4>ACPI Exception (processor_core-0787): AE_NOT_FOUND, Processor Device is not present [20070126]
<6>md: raid1 personality registered for level 1
<6>BIOS EDD facility v0.16 2004-Jun-25, 4 devices found
<6>usbcore: registered new interface driver usbfs
<6>usbcore: registered new interface driver hub
<6>usbcore: registered new device driver usb
<7>ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
<4>ACPI: PCI Interrupt Link [LUB0] enabled at IRQ 21
--
<5>sd 5:0:0:0: [sdd] 490234752 512-byte hardware sectors (251000 MB)
<5>sd 5:0:0:0: [sdd] Write Protect is off
<7>sd 5:0:0:0: [sdd] Mode Sense: 00 3a 00 00
<5>sd 5:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
<6> sdd: sdd1 sdd2 sdd3 sdd4
<5>sd 5:0:0:0: [sdd] Attached SCSI disk
<6>md: raid0 personality registered for level 0
<6>raid5: automatically using best checksumming function: pIII_sse
<4>   pIII_sse  :  5641.000 MB/sec
<4>raid5: using function: pIII_sse (5641.000 MB/sec)
<4>raid6: int32x1    706 MB/s
<4>raid6: int32x2    747 MB/s
<4>raid6: int32x4    671 MB/s
<4>raid6: int32x8    516 MB/s
<4>raid6: mmxx1     1504 MB/s
<4>raid6: mmxx2     2760 MB/s
<4>raid6: sse1x1     344 MB/s
<4>raid6: sse1x2     382 MB/s
<4>raid6: sse2x1     440 MB/s
<4>raid6: sse2x2     640 MB/s
<4>raid6: using algorithm sse2x2 (640 MB/s)
<6>md: raid6 personality registered for level 6
<6>md: raid5 personality registered for level 5
<6>md: raid4 personality registered for level 4
<6>md: md2 stopped.
<6>md: md0 stopped.
<6>md: bind<sdb1>
<6>md: bind<sdc1>
<6>md: bind<sdd1>
<6>md: bind<sda1>
<3>md: md0: raid array is not clean -- starting background reconstruction
<6>raid1: raid set md0 active with 4 out of 4 mirrors
<6>md0: bitmap file is out of date (4 < 5) -- forcing full recovery
<6>md0: bitmap file is out of date, doing full recovery
<6>md0: bitmap initialized from disk: read 11/11 pages, set 327824 bits, status: 0
<6>created bitmap (161 pages) for device md0
<4>swsusp: Basic memory bitmaps created
<4>swsusp: Basic memory bitmaps freed
<4>swsusp: Basic memory bitmaps created
<4>swsusp: Basic memory bitmaps freed
<4>Attempting manual resume
<1>Read-error on swap-device (9:2:8)
<6>md: resync of RAID array md0
<6>md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
<6>md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
<6>md: using 128k window, over a total of 10490340 blocks.
<6>kjournald starting.  Commit interval 5 seconds
<6>EXT3 FS on md0, internal journal
<6>EXT3-fs: mounted filesystem with ordered data mode.
--
<6>md: md1 stopped.
<6>device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel@redhat.com
<6>md: bind<sdb2>
<6>md: bind<sdc2>
<6>md: bind<sdd2>
<6>md: bind<sda2>
<6>raid1: raid set md1 active with 4 out of 4 mirrors
<6>md1: bitmap initialized from disk: read 11/11 pages, set 2 bits, status: 0
<6>created bitmap (161 pages) for device md1
<6>md: md2 stopped.
<6>md: bind<sdb3>
<6>md: bind<sdc3>
<6>md: bind<sdd3>
<6>md: bind<sda3>
<6>raid5: device sda3 operational as raid disk 0
<6>raid5: device sdd3 operational as raid disk 3
<6>raid5: device sdc3 operational as raid disk 2
<6>raid5: device sdb3 operational as raid disk 1
<6>raid5: allocated 4204kB for md2
<4>raid5: raid level 5 set md2 active with 4 out of 4 devices, algorithm 2
<4>RAID5 conf printout:
<4> --- rd:4 wd:4
<4> disk 0, o:1, dev:sda3
<4> disk 1, o:1, dev:sdb3
<4> disk 2, o:1, dev:sdc3
<4> disk 3, o:1, dev:sdd3
<6>md2: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0
--
<6>md: md3 stopped.
<6>md: bind<sdb4>
<7>ieee1394: Host added: ID:BUS[0-00:1023]  GUID[0011d80000916254]
<6>md: bind<sdc4>
<6>md: bind<sdd4>
<6>md: bind<sda4>
<6>raid5: device sda4 operational as raid disk 0
<6>raid5: device sdc4 operational as raid disk 2
<6>raid5: device sdb4 operational as raid disk 1
<6>raid5: allocated 4204kB for md3
<1>raid5: raid level 5 set md3 active with 3 out of 4 devices, algorithm 2
<4>RAID5 conf printout:
<4> --- rd:4 wd:3
<4> disk 0, o:1, dev:sda4
<4> disk 1, o:1, dev:sdb4
<4> disk 2, o:1, dev:sdc4
<4> disk 3, o:1, dev:sdd4
<6>md3: bitmap initialized from disk: read 12/12 pages, set 3 bits, status: 0
--
Loading required kernel modules
doneActivating swap-devices in /etc/fstab...
failedmount: according to mtab, /dev/md0 is already mounted on /

Activating device mapper...
done
Starting MD Raid mdadm: /dev/md1 has been started with 4 drives.
mdadm: /dev/md2 has been started with 4 drives.
mdadm: /dev/md3 has been started with 4 drives.
Checking file systems...
fsck 1.40.2 (12-Jul-2007)
/dev/md1: clean, 74739/5248992 files, 2949291/10486410 blocks
/sbin/fsck.xfs: XFS file system.
-------------------------------------
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help