Assembly failure
flat view
From: Brian Candler <hidden>
Date: 2012-07-10 16:33:45
An odd one here.
Ubuntu 12.04 system, updated to 3.4.0 kernel from the mainline-ppa. Machine
has a boot disk plus 12 other disks in a RAID10 far2 array.
System was working fine, but after most recent reboot mdraid failed to
assemble.
root@dev-storage1:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : inactive sdk[9](S) sdl[10](S) sdj[8](S) sdg[5](S) sdc[1](S) sdm[11](S) sdf[4](S) sde[3](S) sdi[7](S) sdd[2](S) sdb[0](S) sdh[6](S)
35163186720 blocks super 1.2
unused devices: <none>
dmesg shows periodic "export_rdev" messages:
root@dev-storage1:~# dmesg | grep md:
[ 953.986401] md: export_rdev(sdo)
[ 953.988515] md: export_rdev(sdo)
[ 960.237392] md: export_rdev(sdp)
[ 960.241928] md: export_rdev(sdp)
[ 960.965132] md: export_rdev(sdr)
[ 960.967265] md: export_rdev(sdr)
[ 1012.573415] md: export_rdev(sdo)
[ 1012.575650] md: export_rdev(sdo)
[ 1012.829690] md: export_rdev(sdp)
[ 1012.831493] md: export_rdev(sdp)
...
[19378.332473] md: export_rdev(sds)
[19378.333764] md: export_rdev(sds)
[19417.220171] md: export_rdev(sdr)
[19417.221748] md: export_rdev(sdr)
[23739.824227] md: export_rdev(sdr)
[23739.825554] md: export_rdev(sdr)
[23740.568940] md: export_rdev(sds)
[23740.570079] md: export_rdev(sds)
metadata (see below) suggests that some drives think members 1/3/4 are
missing, but those drives think the array is fine. The "Events" counts are
different on some members though.
What's the best thing to do here - attempt to force assembly? Any ideas how
it got into this state?
The machine was rebooted a couple of times but in what should have been a
clean way, i.e. sudo reboot or sudo halt -p.
Many thanks,
Brian.
root@dev-storage1:~# for i in /dev/sd{j..u}; do echo "=== $i ==="; mdadm --examine $i; done
=== /dev/sdj ===
/dev/sdj:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e96965d5:bf8986b7:fa83b813:e27aa17f
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 10 10:46:19 2012
Checksum : 5673ac95 - correct
Events : 29374
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 8
Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
=== /dev/sdk ===
/dev/sdk:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : f15bb47f:85ca59b1:cad42dec:f8b1b63c
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 10 10:46:19 2012
Checksum : 9020674a - correct
Events : 29374
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 9
Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
=== /dev/sdl ===
/dev/sdl:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e2d82a4c:8409d883:cf2d9b7c:83829aad
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 10 10:46:19 2012
Checksum : 30664ccb - correct
Events : 29374
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 10
Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
=== /dev/sdm ===
/dev/sdm:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e3603570:8e767487:63f3131b:afe358ea
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 10 10:46:19 2012
Checksum : 33446897 - correct
Events : 29374
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 11
Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
=== /dev/sdn ===
/dev/sdn:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 549a5230:005cedf8:b37a0d7e:36648ff0
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 10 10:46:19 2012
Checksum : ce0a8f46 - correct
Events : 29374
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 0
Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
=== /dev/sdo ===
/dev/sdo:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : b09f3869:09ce8a89:31ed7097:d3621064
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Jul 7 14:02:07 2012
Checksum : 246eb119 - correct
Events : 29355
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 2
Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
=== /dev/sdp ===
/dev/sdp:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 71494ee2:1504b35a:00a1d927:543db7c6
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Jul 7 14:00:55 2012
Checksum : 61c11eeb - correct
Events : 29352
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 3
Array State : A.AA.AAAAAAA ('A' == active, '.' == missing)
=== /dev/sdq ===
/dev/sdq:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 17fcd5b8:97fc715f:0877d022:3770d08b
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 10 10:46:19 2012
Checksum : fd1699fc - correct
Events : 29374
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 7
Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
=== /dev/sdr ===
/dev/sdr:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e0565412:a68cf236:9da9a141:e89f935a
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 3 04:21:50 2012
Checksum : 1ba72ec1 - correct
Events : 20228
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 1
Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
=== /dev/sds ===
/dev/sds:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : c0f3bbab:70f3e69e:1314cdca:072d5ad9
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 3 06:37:33 2012
Checksum : 24f36d4c - correct
Events : 29312
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 5
Array State : A.AA.AAAAAAA ('A' == active, '.' == missing)
=== /dev/sdt ===
/dev/sdt:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 6b4f9595:4de46aa8:fa695fe8:59b797e7
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 10 10:46:19 2012
Checksum : 44250f1b - correct
Events : 29374
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 6
Array State : A.A..AAAAAAA ('A' == active, '.' == missing)
=== /dev/sdu ===
/dev/sdu:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 16b260fd:e49bd157:da886cd0:5394e194
Name : storage1:storage1
Creation Time : Thu Jun 7 13:51:21 2012
Raid Level : raid10
Raid Devices : 12
Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
Array Size : 35163168768 (16767.11 GiB 18003.54 GB)
Used Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 46f03854:de80bec8:b44b062c:dd265ba3
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jul 3 06:36:18 2012
Checksum : 22812848 - correct
Events : 29272
Layout : far=2
Chunk Size : 1024K
Device Role : Active device 4
Array State : A.AAAAAAAAAA ('A' == active, '.' == missing)