Re: Failed --grow. Recovery possible?
From: Stephan Stachurski <hidden>
Date: 2010-03-24 04:46:05
Neil gave me these four steps to follow: What I suggest you do is: 1/ find the backup of the first 1152K 2/ re-create the array as the original 6-drive raid5 3/ Check if the backup needs to be restored and possibly restore it 4/ Don't use the new drives until you are really sure they will work. The only thing I am sure of is number 4 that the new drives work. I have tested them individually in the same controller and they're working fine. 1 - "You would need to look at the code in Grow.c to see where it is written" The last time I have looked at C code was over ten years ago when I was 15 years old. I wish I would have become a kernel hacker instead of a lowly web developer, but this isn't my area of expertise. Grow.c is a huge file it's quite daunting. I don't even know what I could hope to learn from this data once I find it. 2 is not quite easy. I've re-created the original array (in the order that I'm 99% certain the original array was) and no attempt to mount or check a file system there will work. There must be something between step 1 and 2 where I have to restore the "critical section" to a certain spot, maybe? On Tue, Mar 23, 2010 at 5:07 PM, Michael Evans [off-list ref] wrote:
On Tue, Mar 23, 2010 at 12:53 PM, Stephan Stachurski [off-list ref] wrote:quoted
Sorry, it looks like I sent two replies directly to NeilBrown instead of the mailing list. Here they are: First reply: Before I checked this email, I upgraded to the newest version of mdadm on advice I got from #linux on freenode. The array is no longer segfaults when it's assembled. Instead it picks up exactly where it left off trying to grow the array but not actually progressing. In the new version of mdadm, however, after a short while the drives on the mv_sas controller are dropped and mdstat reports that a resync is pending. This looks like an improvement to me. I am going to try to test out the controller and drives to see if I can find out what's going on. ---------------------- Second reply: I hope I haven't screwed up my disks beyond saving. I was using this earlier question on the mailing list as a reference http://www.mail-archive.com/linux-raid@vger.kernel.org/msg08907.html . If I could re-assemble the original array, I would feel a lot more comfortable proceeding. In the above referenced thread, Greg Nicholson mentions that order matters when it comes to assembling arrays, so I wrote a perl script (full of terrible hacks) that took the list of the 6 original devices and iterated over the permutations of the order of those devices, assembled the array with --assume-clean and attempt to mount the file system. This failed for all 720 permutations. It was probably a stupid idea, anyway... I also tried plugging in a known working disk into the mv_sas controller that may have had an issue and it looked like everything was working OK. Now I'm really not sure what to do. I'm completely lost. Thanks again for your help. On Fri, Mar 19, 2010 at 12:02 AM, Neil Brown [off-list ref] wrote:quoted
On Thu, 18 Mar 2010 21:52:54 -0400 Stephan Stachurski [off-list ref] wrote:quoted
I have had a RAID5 up for quite a while with 6 disks. I recently added 4 and attempted to grow the array to span all 10 devices. For an hour after starting the grow command, the speed of the operation was 0K/s the entire time. I thought something must be wrong, and the best course of action would be to reboot and start over from a clean boot. I'm not a linux expert so I thought that if I rebooted everything would try to exit gracefully. After one hour, the system still had not finished shutting down. I then did alt-sysreq RSEISUB waiting over one minute between each command. I've included the syslog of what happened up until the next start up, but put it last because it's by far the longest.I think you will be able to get your data back. It won't be trivial, but it should be possible. I looks like the driver for the mv_sas controller has issues. When md/raid5 started writing data on to them to reshape the array something went wrong and the writes didn't complete, so nothing else happened. I don't know why mdadm is getting a segmentation fault. Possibly this is fixed in a newer version of mdadm. However it is possibly good that it didn't manage to restart the array fully as it would probably has just failed again and might have made more of a mess. To get you data back we need to understand exactly what happened. What should happen when you run "mdadm --grow ... " is that it sets up for a reshape but doesn't let it progress. Then it prints: mdadm: Need to backup 1152K of critical section.. It then copies the first 1152K (in the case of 6->10 with 256K chunk) from the start of the array to near the end of each of the 'spares'. Then it allows the reshape to proceed. Once the reshape has progressed past that 1152K it removes the copy that it made (erases some metadata for it) and prints mdadm: ... critical section passed. I presume that it didn't successfully pass the critical section, else the Reshape Position would be greater than 0. It is possible that the reshape didn't start at all and your data is exactly where you left it, but we cannot be sure without looking... What I suggest you do is: 1/ find the backup of the first 1152K 2/ re-create the array as the original 6-drive raid5 3/ Check if the backup needs to be restored and possibly restore it 4/ Don't use the new drives until you are really sure they will work. 1 is the hardest. I have a vague plan of giving mdadm the ability to do this but I haven't yet. I could possibly do it next week some time if you can wait. You would need to look at the code in Grow.c to see where it is written. I think there is a block of metadata near the end of the device - just before the md metadata - which records what has been backed up where. Once you find and decode that from one of the spares you can easily use 'dd' to extract the backup. 2 is quite easy: mdadm -C /dev/md0 -e 0.90 -l 5 -n 6 -c 256 \ --assume-clean /dev/sdb /dev/sdg /dev/sdh /dev/sdd /dev/sda /dev/sdc Make sure you have the devices if the right order. If you aren't sure, then mdadm -E list..of..devices | grep this should give you an ascending series in columns 2 and 5. 3 is simply a 'cmp' between /dev/md0 and the backup that you restored. or maybe just 'fsck' of /dev/md0. If you decide to restore (be sure before you do), just dd the backup to the start of /dev/md0 I don't know how you can make yourself sure that the drives really do work. Lots of testing of the new devices by themselves in an array ??? Good luck. NeilBrownquoted
When I rebooted, the array seemed to be up, but mounting it resulted in a bad FS type error, even when I tried to specify it (ext4). After stopping the inactive array, and trying to reassemble it, mdadm crashed to segmentation fault. Is it possible to recover the data? We have backups, but they're spread out over 1500 DVDs. When I examine the drives, the output looks pretty much like this for each drive (6 drives say active and 4 say clean, corresponding to the 6 original and 4 added drives): $ mdadm --examine /dev/sda /dev/sda: Magic : a92b4efc Version : 00.91.00 UUID : 56c16545:07db76d6:e368bf24:bd0fce41 Creation Time : Tue Feb 2 09:58:58 2010 Raid Level : raid5 Used Dev Size : 976762368 (931.51 GiB 1000.20 GB) Array Size : 8790861312 (8383.62 GiB 9001.84 GB) Raid Devices : 10 Total Devices : 10 Preferred Minor : 0 Reshape pos'n : 0 Delta Devices : 4 (6->10) Update Time : Thu Mar 18 23:33:40 2010 State : active Active Devices : 10 Working Devices : 10 Failed Devices : 0 Spare Devices : 0 Checksum : 79904299 - correct Events : 270611 Layout : left-symmetric Chunk Size : 256K Number Major Minor RaidDevice State this 4 8 0 4 active sync /dev/sda 0 0 8 16 0 active sync /dev/sdb 1 1 8 96 1 active sync /dev/sdg 2 2 8 112 2 active sync /dev/sdh 3 3 8 48 3 active sync /dev/sdd 4 4 8 0 4 active sync /dev/sda 5 5 8 32 5 active sync /dev/sdc 6 6 8 160 6 active sync /dev/sdk 7 7 8 144 7 active sync /dev/sdj 8 8 8 128 8 active sync /dev/sdi 9 9 8 80 9 active sync /dev/sdf #syslog showing how mdadm detects the bad 10-drive array on boot Mar 18 23:47:35 raidserver kernel: [ 2.178393] md: bind<sdc> Mar 18 23:47:35 raidserver kernel: [ 2.193132] md: bind<sdd> Mar 18 23:47:35 raidserver kernel: [ 2.211906] md: bind<sdb> Mar 18 23:47:35 raidserver kernel: [ 2.220062] md: bind<sda> Mar 18 23:47:35 raidserver kernel: [ 2.230062] ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[22] MMIO=[fd6ff000-fd6ff7ff] Max Packet=[2048] IR/IT contexts=[4/8] Mar 18 23:47:35 raidserver kernel: [ 3.551483] ieee1394: Host added: ID:BUS[0-00:1023] GUID[003635c7006cf049] Mar 18 23:47:35 raidserver kernel: [ 6.579349] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 0 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 6.579353] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 0 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 6.780038] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 1 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 6.780041] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 1 attach sas addr is 1 Mar 18 23:47:35 raidserver kernel: [ 6.990054] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 2 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 6.990057] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 2 attach sas addr is 2 Mar 18 23:47:35 raidserver kernel: [ 7.200052] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 3 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 7.200055] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 3 attach sas addr is 3 Mar 18 23:47:35 raidserver kernel: [ 7.310035] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 4 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 7.310038] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 4 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 7.420035] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 5 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 7.420038] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 5 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 7.630052] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 6 attach dev info is 2000000 Mar 18 23:47:35 raidserver kernel: [ 7.630055] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 6 attach sas addr is 6 Mar 18 23:47:35 raidserver kernel: [ 7.840053] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 7 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 7.840056] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 7 attach sas addr is 7 Mar 18 23:47:35 raidserver kernel: [ 7.840062] scsi8 : mvsas Mar 18 23:47:35 raidserver kernel: [ 7.840605] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 0 byte dmaded. Mar 18 23:47:35 raidserver kernel: [ 7.840610] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 1 byte dmaded. Mar 18 23:47:35 raidserver kernel: [ 7.840614] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 2 byte dmaded. Mar 18 23:47:35 raidserver kernel: [ 7.840617] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 3 byte dmaded. Mar 18 23:47:35 raidserver kernel: [ 7.840621] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 6 byte dmaded. Mar 18 23:47:35 raidserver kernel: [ 7.840624] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 7 byte dmaded. Mar 18 23:47:35 raidserver kernel: [ 7.840871] mvsas 0000:03:00.0: mvsas: driver version 0.8.2 Mar 18 23:47:35 raidserver kernel: [ 7.840885] mvsas 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 Mar 18 23:47:35 raidserver kernel: [ 7.840891] mvsas 0000:03:00.0: setting latency timer to 64 Mar 18 23:47:35 raidserver kernel: [ 7.841965] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found dev[0:5] is gone. Mar 18 23:47:35 raidserver kernel: [ 7.843312] ata9.00: ATA-8: WDC WD10EARS-00Y5B1, 80.00A80, max UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.843316] ata9.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) Mar 18 23:47:35 raidserver kernel: [ 7.843480] mvsas 0000:03:00.0: mvsas: PCI-E x4, Bandwidth Usage: 2.5 Gbps Mar 18 23:47:35 raidserver kernel: [ 7.845117] ata9.00: configured for UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.845182] scsi 8:0:0:0: Direct-Access ATA WDC WD10EARS-00Y 80.0 PQ: 0 ANSI: 5 Mar 18 23:47:35 raidserver kernel: [ 7.845801] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found dev[1:5] is gone. Mar 18 23:47:35 raidserver kernel: [ 7.846588] ata10.00: ATA-8: WDC WD10EADS-00L5B1, 01.01A01, max UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.846591] ata10.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) Mar 18 23:47:35 raidserver kernel: [ 7.847419] ata10.00: configured for UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.847455] scsi 8:0:1:0: Direct-Access ATA WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 5 Mar 18 23:47:35 raidserver kernel: [ 7.848069] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found dev[2:5] is gone. Mar 18 23:47:35 raidserver kernel: [ 7.848894] ata11.00: ATA-8: WDC WD10EACS-00D6B1, 01.01A01, max UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.848897] ata11.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) Mar 18 23:47:35 raidserver kernel: [ 7.849713] ata11.00: configured for UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.849751] scsi 8:0:2:0: Direct-Access ATA WDC WD10EACS-00D 01.0 PQ: 0 ANSI: 5 Mar 18 23:47:35 raidserver kernel: [ 7.850909] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found dev[3:5] is gone. Mar 18 23:47:35 raidserver kernel: [ 7.852188] ata12.00: ATA-8: WDC WD10EARS-00Y5B1, 80.00A80, max UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.852192] ata12.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) Mar 18 23:47:35 raidserver kernel: [ 7.853488] ata12.00: configured for UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.853524] scsi 8:0:3:0: Direct-Access ATA WDC WD10EARS-00Y 80.0 PQ: 0 ANSI: 5 Mar 18 23:47:35 raidserver kernel: [ 7.854665] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found dev[4:5] is gone. Mar 18 23:47:35 raidserver kernel: [ 7.855955] ata13.00: ATA-8: WDC WD10EARS-00Y5B1, 80.00A80, max UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.855959] ata13.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) Mar 18 23:47:35 raidserver kernel: [ 7.857258] ata13.00: configured for UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.857293] scsi 8:0:4:0: Direct-Access ATA WDC WD10EARS-00Y 80.0 PQ: 0 ANSI: 5 Mar 18 23:47:35 raidserver kernel: [ 7.858437] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found dev[5:5] is gone. Mar 18 23:47:35 raidserver kernel: [ 7.859713] ata14.00: ATA-8: WDC WD10EARS-00Y5B1, 80.00A80, max UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.859716] ata14.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) Mar 18 23:47:35 raidserver kernel: [ 7.861014] ata14.00: configured for UDMA/133 Mar 18 23:47:35 raidserver kernel: [ 7.861051] scsi 8:0:5:0: Direct-Access ATA WDC WD10EARS-00Y 80.0 PQ: 0 ANSI: 5 Mar 18 23:47:35 raidserver kernel: [ 8.831403] sd 8:0:0:0: Attached scsi generic sg5 type 0 Mar 18 23:47:35 raidserver kernel: [ 8.831507] sd 8:0:1:0: Attached scsi generic sg6 type 0 Mar 18 23:47:35 raidserver kernel: [ 8.831610] sd 8:0:2:0: Attached scsi generic sg7 type 0 Mar 18 23:47:35 raidserver kernel: [ 8.831713] sd 8:0:3:0: Attached scsi generic sg8 type 0 Mar 18 23:47:35 raidserver kernel: [ 8.831823] sd 8:0:4:0: Attached scsi generic sg9 type 0 Mar 18 23:47:35 raidserver kernel: [ 8.831927] sd 8:0:5:0: Attached scsi generic sg10 type 0 Mar 18 23:47:35 raidserver kernel: [ 8.832481] sd 8:0:0:0: [sdf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) Mar 18 23:47:35 raidserver kernel: [ 8.832485] sd 8:0:1:0: [sdg] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) Mar 18 23:47:35 raidserver kernel: [ 8.832547] sd 8:0:1:0: [sdg] Write Protect is off Mar 18 23:47:35 raidserver kernel: [ 8.832551] sd 8:0:0:0: [sdf] Write Protect is off Mar 18 23:47:35 raidserver kernel: [ 8.832555] sd 8:0:0:0: [sdf] Mode Sense: 00 3a 00 00 Mar 18 23:47:35 raidserver kernel: [ 8.832559] sd 8:0:1:0: [sdg] Mode Sense: 00 3a 00 00 Mar 18 23:47:35 raidserver kernel: [ 8.832586] sd 8:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Mar 18 23:47:35 raidserver kernel: [ 8.832590] sd 8:0:1:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Mar 18 23:47:35 raidserver kernel: [ 8.832806] sdg: Mar 18 23:47:35 raidserver kernel: [ 8.832853] sdf: Mar 18 23:47:35 raidserver kernel: [ 8.832914] sd 8:0:2:0: [sdh] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) Mar 18 23:47:35 raidserver kernel: [ 8.832953] sd 8:0:2:0: [sdh] Write Protect is off Mar 18 23:47:35 raidserver kernel: [ 8.832956] sd 8:0:2:0: [sdh] Mode Sense: 00 3a 00 00 Mar 18 23:47:35 raidserver kernel: [ 8.832976] sd 8:0:2:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Mar 18 23:47:35 raidserver kernel: [ 8.833087] sdh: Mar 18 23:47:35 raidserver kernel: [ 8.833182] sd 8:0:3:0: [sdi] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) Mar 18 23:47:35 raidserver kernel: [ 8.833197] sd 8:0:4:0: [sdj] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) Mar 18 23:47:35 raidserver kernel: [ 8.833238] sd 8:0:3:0: [sdi] Write Protect is off Mar 18 23:47:35 raidserver kernel: [ 8.833241] sd 8:0:3:0: [sdi] Mode Sense: 00 3a 00 00 Mar 18 23:47:35 raidserver kernel: [ 8.833250] sd 8:0:4:0: [sdj] Write Protect is off Mar 18 23:47:35 raidserver kernel: [ 8.833252] sd 8:0:4:0: [sdj] Mode Sense: 00 3a 00 00 Mar 18 23:47:35 raidserver kernel: [ 8.833270] sd 8:0:3:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Mar 18 23:47:35 raidserver kernel: [ 8.833279] sd 8:0:4:0: [sdj] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Mar 18 23:47:35 raidserver kernel: [ 8.833442] sdi: Mar 18 23:47:35 raidserver kernel: [ 8.833467] sdj: Mar 18 23:47:35 raidserver kernel: [ 8.833553] sd 8:0:5:0: [sdk] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) Mar 18 23:47:35 raidserver kernel: [ 8.833593] sd 8:0:5:0: [sdk] Write Protect is off Mar 18 23:47:35 raidserver kernel: [ 8.833596] sd 8:0:5:0: [sdk] Mode Sense: 00 3a 00 00 Mar 18 23:47:35 raidserver kernel: [ 8.833617] sd 8:0:5:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Mar 18 23:47:35 raidserver kernel: [ 8.833734] sdk: unknown partition table Mar 18 23:47:35 raidserver kernel: [ 8.846211] sd 8:0:2:0: [sdh] Attached SCSI disk Mar 18 23:47:35 raidserver kernel: [ 8.846599] unknown partition table Mar 18 23:47:35 raidserver kernel: [ 8.846759] sd 8:0:1:0: [sdg] Attached SCSI disk Mar 18 23:47:35 raidserver kernel: [ 9.316040] unknown partition table Mar 18 23:47:35 raidserver kernel: [ 9.316243] sd 8:0:3:0: [sdi] Attached SCSI disk Mar 18 23:47:35 raidserver kernel: [ 9.316249] unknown partition table Mar 18 23:47:35 raidserver kernel: [ 9.316404] sd 8:0:0:0: [sdf] Attached SCSI disk Mar 18 23:47:35 raidserver kernel: [ 9.317860] unknown partition table Mar 18 23:47:35 raidserver kernel: [ 9.318033] sd 8:0:4:0: [sdj] Attached SCSI disk Mar 18 23:47:35 raidserver kernel: [ 9.321964] unknown partition table Mar 18 23:47:35 raidserver kernel: [ 9.322127] sd 8:0:5:0: [sdk] Attached SCSI disk Mar 18 23:47:35 raidserver kernel: [ 12.220036] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 0 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 12.220039] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 0 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 12.330034] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 1 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 12.330037] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 1 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 12.440035] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 2 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 12.440037] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 2 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 12.550034] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 3 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 12.550038] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 3 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 12.660035] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 4 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 12.660038] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 4 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 12.770035] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 5 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 12.770037] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 5 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 12.880035] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 6 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 12.880037] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 6 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 12.990034] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 7 attach dev info is 0 Mar 18 23:47:35 raidserver kernel: [ 12.990037] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 7 attach sas addr is 0 Mar 18 23:47:35 raidserver kernel: [ 12.990043] scsi9 : mvsas Mar 18 23:47:35 raidserver kernel: [ 13.595116] md: bind<sdi> Mar 18 23:47:35 raidserver kernel: [ 13.651656] md: bind<sdf> Mar 18 23:47:35 raidserver kernel: [ 13.653928] md: bind<sdj> Mar 18 23:47:35 raidserver kernel: [ 13.854601] md: bind<sdk> Mar 18 23:47:35 raidserver kernel: [ 14.055322] md: bind<sdh> Mar 18 23:47:35 raidserver kernel: [ 14.255683] md: bind<sdg> Mar 18 23:47:35 raidserver kernel: [ 14.259239] xor: automatically using best checksumming function: generic_sse Mar 18 23:47:35 raidserver kernel: [ 14.300015] generic_sse: 6593.600 MB/sec Mar 18 23:47:35 raidserver kernel: [ 14.300018] xor: using function: generic_sse (6593.600 MB/sec) Mar 18 23:47:35 raidserver kernel: [ 14.300599] async_tx: api initialized (async) Mar 18 23:47:35 raidserver kernel: [ 14.470026] raid6: int64x1 1711 MB/s Mar 18 23:47:35 raidserver kernel: [ 14.640019] raid6: int64x2 2392 MB/s Mar 18 23:47:35 raidserver kernel: [ 14.810046] raid6: int64x4 1567 MB/s Mar 18 23:47:35 raidserver kernel: [ 14.980047] raid6: int64x8 1540 MB/s Mar 18 23:47:35 raidserver kernel: [ 15.150016] raid6: sse2x1 2931 MB/s Mar 18 23:47:35 raidserver kernel: [ 15.320030] raid6: sse2x2 3916 MB/s Mar 18 23:47:35 raidserver kernel: [ 15.490023] raid6: sse2x4 4088 MB/s Mar 18 23:47:35 raidserver kernel: [ 15.490025] raid6: using algorithm sse2x4 (4088 MB/s) Mar 18 23:47:35 raidserver kernel: [ 15.493236] md: raid6 personality registered for level 6 Mar 18 23:47:35 raidserver kernel: [ 15.493240] md: raid5 personality registered for level 5 Mar 18 23:47:35 raidserver kernel: [ 15.493242] md: raid4 personality registered for level 4 Mar 18 23:47:35 raidserver kernel: [ 15.493642] raid5: md0 is not clean -- starting background reconstruction Mar 18 23:47:35 raidserver kernel: [ 15.493645] raid5: reshape_position too early for auto-recovery - aborting. Mar 18 23:47:35 raidserver kernel: [ 15.493647] md: pers->run() failed ... Mar 18 23:47:35 raidserver kernel: [ 15.931347] md: linear personality registered for level -1 Mar 18 23:47:35 raidserver kernel: [ 15.934321] md: multipath personality registered for level -4 Mar 18 23:47:35 raidserver kernel: [ 15.936772] md: raid0 personality registered for level 0 Mar 18 23:47:35 raidserver kernel: [ 15.940395] md: raid1 personality registered for level 1 Mar 18 23:47:35 raidserver kernel: [ 15.949846] md: raid10 personality registered for level 10 #syslog of the segfault Mar 18 23:41:47 raidserver kernel: [ 155.028406] md: md0 stopped. Mar 18 23:41:47 raidserver kernel: [ 155.028443] md: unbind<sdg> Mar 18 23:41:47 raidserver kernel: [ 155.051309] md: export_rdev(sdg) Mar 18 23:41:47 raidserver kernel: [ 155.051464] md: unbind<sdk> Mar 18 23:41:47 raidserver kernel: [ 155.091288] md: export_rdev(sdk) Mar 18 23:41:47 raidserver kernel: [ 155.091418] md: unbind<sdj> Mar 18 23:41:47 raidserver kernel: [ 155.131274] md: export_rdev(sdj) Mar 18 23:41:47 raidserver kernel: [ 155.131407] md: unbind<sdi> Mar 18 23:41:47 raidserver kernel: [ 155.161277] md: export_rdev(sdi) Mar 18 23:41:47 raidserver kernel: [ 155.161421] md: unbind<sdl> Mar 18 23:41:47 raidserver kernel: [ 155.191275] md: export_rdev(sdl) Mar 18 23:41:47 raidserver kernel: [ 155.191400] md: unbind<sdh> Mar 18 23:41:47 raidserver kernel: [ 155.221276] md: export_rdev(sdh) Mar 18 23:41:47 raidserver kernel: [ 155.221403] md: unbind<sda> Mar 18 23:41:47 raidserver kernel: [ 155.251276] md: export_rdev(sda) Mar 18 23:41:47 raidserver kernel: [ 155.251385] md: unbind<sdb> Mar 18 23:41:47 raidserver kernel: [ 155.281277] md: export_rdev(sdb) Mar 18 23:41:47 raidserver kernel: [ 155.281379] md: unbind<sdc> Mar 18 23:41:47 raidserver kernel: [ 155.311276] md: export_rdev(sdc) Mar 18 23:41:47 raidserver kernel: [ 155.311377] md: unbind<sdd> Mar 18 23:41:47 raidserver mdadm[1738]: DeviceDisappeared event detected on md device /dev/md0 Mar 18 23:41:47 raidserver kernel: [ 155.341274] md: export_rdev(sdd) Mar 18 23:43:46 raidserver kernel: [ 274.246878] md: md0 stopped. Mar 18 23:45:17 raidserver kernel: [ 365.227246] md: md0 stopped. Mar 18 23:45:18 raidserver kernel: [ 365.828455] __ratelimit: 30 callbacks suppressed Mar 18 23:45:18 raidserver kernel: [ 365.828466] mdadm[2874]: segfault at 38 ip 00000000004184ff sp 00007fffc2aa6bd0 error 4 in mdadm[400000+2a000] #syslog of the grow and subsequent reboot Mar 18 23:27:41 raidserver kernel: [ 861.411806] md: bind<sdf> Mar 18 23:27:42 raidserver ata_id[2638]: HDIO_GET_IDENTITY failed for '/dev/sdi' Mar 18 23:27:42 raidserver kernel: [ 862.024028] md: bind<sdi> Mar 18 23:27:43 raidserver ata_id[2650]: HDIO_GET_IDENTITY failed for '/dev/sdj' Mar 18 23:27:43 raidserver kernel: [ 863.133531] md: bind<sdj> Mar 18 23:27:43 raidserver ata_id[2658]: HDIO_GET_IDENTITY failed for '/dev/sdk' Mar 18 23:27:43 raidserver kernel: [ 863.285276] md: bind<sdk> Mar 18 23:28:35 raidserver kernel: [ 914.792375] RAID5 conf printout: Mar 18 23:28:35 raidserver kernel: [ 914.792385] --- rd:10 wd:10 Mar 18 23:28:35 raidserver kernel: [ 914.792393] disk 0, o:1, dev:sdb Mar 18 23:28:35 raidserver kernel: [ 914.792398] disk 1, o:1, dev:sdg Mar 18 23:28:35 raidserver kernel: [ 914.792402] disk 2, o:1, dev:sdh Mar 18 23:28:35 raidserver kernel: [ 914.792406] disk 3, o:1, dev:sdd Mar 18 23:28:35 raidserver kernel: [ 914.792410] disk 4, o:1, dev:sda Mar 18 23:28:35 raidserver kernel: [ 914.792415] disk 5, o:1, dev:sdc Mar 18 23:28:35 raidserver kernel: [ 914.792419] disk 6, o:1, dev:sdk Mar 18 23:28:35 raidserver kernel: [ 914.792443] RAID5 conf printout: Mar 18 23:28:35 raidserver kernel: [ 914.792447] --- rd:10 wd:10 Mar 18 23:28:35 raidserver kernel: [ 914.792450] disk 0, o:1, dev:sdb Mar 18 23:28:35 raidserver kernel: [ 914.792454] disk 1, o:1, dev:sdg Mar 18 23:28:35 raidserver kernel: [ 914.792457] disk 2, o:1, dev:sdh Mar 18 23:28:35 raidserver kernel: [ 914.792461] disk 3, o:1, dev:sdd Mar 18 23:28:35 raidserver kernel: [ 914.792465] disk 4, o:1, dev:sda Mar 18 23:28:35 raidserver kernel: [ 914.792469] disk 5, o:1, dev:sdc Mar 18 23:28:35 raidserver kernel: [ 914.792472] disk 6, o:1, dev:sdk Mar 18 23:28:35 raidserver kernel: [ 914.792476] disk 7, o:1, dev:sdj Mar 18 23:28:35 raidserver kernel: [ 914.792484] RAID5 conf printout: Mar 18 23:28:35 raidserver kernel: [ 914.792487] --- rd:10 wd:10 Mar 18 23:28:35 raidserver kernel: [ 914.792491] disk 0, o:1, dev:sdb Mar 18 23:28:35 raidserver kernel: [ 914.792494] disk 1, o:1, dev:sdg Mar 18 23:28:35 raidserver kernel: [ 914.792498] disk 2, o:1, dev:sdh Mar 18 23:28:35 raidserver kernel: [ 914.792502] disk 3, o:1, dev:sdd Mar 18 23:28:35 raidserver kernel: [ 914.792506] disk 4, o:1, dev:sda Mar 18 23:28:35 raidserver kernel: [ 914.792509] disk 5, o:1, dev:sdc Mar 18 23:28:35 raidserver kernel: [ 914.792513] disk 6, o:1, dev:sdk Mar 18 23:28:35 raidserver kernel: [ 914.792517] disk 7, o:1, dev:sdj Mar 18 23:28:35 raidserver kernel: [ 914.792520] disk 8, o:1, dev:sdi Mar 18 23:28:35 raidserver kernel: [ 914.792528] RAID5 conf printout: Mar 18 23:28:35 raidserver kernel: [ 914.792531] --- rd:10 wd:10 Mar 18 23:28:35 raidserver kernel: [ 914.792535] disk 0, o:1, dev:sdb Mar 18 23:28:35 raidserver kernel: [ 914.792538] disk 1, o:1, dev:sdg Mar 18 23:28:35 raidserver kernel: [ 914.792542] disk 2, o:1, dev:sdh Mar 18 23:28:35 raidserver kernel: [ 914.792545] disk 3, o:1, dev:sdd Mar 18 23:28:35 raidserver kernel: [ 914.792549] disk 4, o:1, dev:sda Mar 18 23:28:35 raidserver kernel: [ 914.792552] disk 5, o:1, dev:sdc Mar 18 23:28:35 raidserver kernel: [ 914.792556] disk 6, o:1, dev:sdk Mar 18 23:28:35 raidserver kernel: [ 914.792559] disk 7, o:1, dev:sdj Mar 18 23:28:35 raidserver kernel: [ 914.792563] disk 8, o:1, dev:sdi Mar 18 23:28:35 raidserver kernel: [ 914.792567] disk 9, o:1, dev:sdf Mar 18 23:28:35 raidserver kernel: [ 914.792713] md: reshape of RAID array md0 Mar 18 23:28:35 raidserver kernel: [ 914.792722] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. Mar 18 23:28:35 raidserver kernel: [ 914.792728] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape. Mar 18 23:28:35 raidserver kernel: [ 914.792746] md: using 128k window, over a total of 976762368 blocks. Mar 18 23:28:35 raidserver mdadm[1627]: RebuildStarted event detected on md device /dev/md0 Mar 18 23:28:35 raidserver mdadm[1627]: SpareActive event detected on md device /dev/md0, component device /dev/sdk Mar 18 23:28:35 raidserver mdadm[1627]: SpareActive event detected on md device /dev/md0, component device /dev/sdj Mar 18 23:28:35 raidserver mdadm[1627]: SpareActive event detected on md device /dev/md0, component device /dev/sdi Mar 18 23:28:35 raidserver mdadm[1627]: SpareActive event detected on md device /dev/md0, component device /dev/sdf Mar 18 23:29:05 raidserver kernel: [ 945.010492] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:29:05 raidserver kernel: [ 945.010501] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:29:05 raidserver kernel: [ 945.010517] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:29:05 raidserver kernel: [ 945.010523] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:29:36 raidserver kernel: [ 976.010049] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:29:36 raidserver kernel: [ 976.010058] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:29:36 raidserver kernel: [ 976.010071] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:29:36 raidserver kernel: [ 976.010077] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:30:07 raidserver kernel: [ 1007.010495] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:30:07 raidserver kernel: [ 1007.010504] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:30:07 raidserver kernel: [ 1007.010518] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:30:07 raidserver kernel: [ 1007.010525] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:30:38 raidserver kernel: [ 1038.010051] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:30:38 raidserver kernel: [ 1038.010060] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:30:38 raidserver kernel: [ 1038.010075] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:30:38 raidserver kernel: [ 1038.010081] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:31:09 raidserver kernel: [ 1069.010493] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:31:09 raidserver kernel: [ 1069.010503] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:31:09 raidserver kernel: [ 1069.010517] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:31:09 raidserver kernel: [ 1069.010523] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:31:20 raidserver kernel: [ 1080.620078] INFO: task md0_reshape:2679 blocked for more than 120 seconds. Mar 18 23:31:20 raidserver kernel: [ 1080.620086] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 18 23:31:20 raidserver kernel: [ 1080.620092] md0_reshape D 00000000ffffffff 0 2679 2 0x00000000 Mar 18 23:31:20 raidserver kernel: [ 1080.620103] ffff8800441e1ad0 0000000000000046 ffff8800441e1a80 0000000000015880 Mar 18 23:31:20 raidserver kernel: [ 1080.620113] ffff880068199a60 0000000000015880 0000000000015880 0000000000015880 Mar 18 23:31:20 raidserver kernel: [ 1080.620122] 0000000000015880 ffff880068199a60 0000000000015880 0000000000015880 Mar 18 23:31:20 raidserver kernel: [ 1080.620131] Call Trace: Mar 18 23:31:20 raidserver kernel: [ 1080.620169] [<ffffffffa01d6ce1>] get_active_stripe+0x2a1/0x360 [raid456] Mar 18 23:31:20 raidserver kernel: [ 1080.620185] [<ffffffff81053d60>] ? default_wake_function+0x0/0x10 Mar 18 23:31:20 raidserver kernel: [ 1080.620197] [<ffffffffa01d92e0>] reshape_request+0x4a0/0x980 [raid456] Mar 18 23:31:20 raidserver kernel: [ 1080.620210] [<ffffffffa01d9ada>] sync_request+0x31a/0x3a0 [raid456] Mar 18 23:31:20 raidserver kernel: [ 1080.620221] [<ffffffffa01d69ae>] ? raid5_unplug_device+0x7e/0x110 [raid456] Mar 18 23:31:20 raidserver kernel: [ 1080.620233] [<ffffffff813daa1e>] md_do_sync+0x5fe/0xba0 Mar 18 23:31:20 raidserver kernel: [ 1080.620242] [<ffffffff813db774>] md_thread+0x44/0x120 Mar 18 23:31:20 raidserver kernel: [ 1080.620249] [<ffffffff813db730>] ? md_thread+0x0/0x120 Mar 18 23:31:20 raidserver kernel: [ 1080.620257] [<ffffffff81078746>] kthread+0xa6/0xb0 Mar 18 23:31:20 raidserver kernel: [ 1080.620266] [<ffffffff810130ea>] child_rip+0xa/0x20 Mar 18 23:31:20 raidserver kernel: [ 1080.620273] [<ffffffff810786a0>] ? kthread+0x0/0xb0 Mar 18 23:31:20 raidserver kernel: [ 1080.620279] [<ffffffff810130e0>] ? child_rip+0x0/0x20 Mar 18 23:31:40 raidserver kernel: [ 1100.010040] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:31:40 raidserver kernel: [ 1100.010049] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:31:40 raidserver kernel: [ 1100.010065] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:31:40 raidserver kernel: [ 1100.010072] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:31:40 raidserver kernel: [ 1100.010127] sd 8:0:1:0: [sdg] Unhandled error code Mar 18 23:31:40 raidserver kernel: [ 1100.010132] sd 8:0:1:0: [sdg] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT Mar 18 23:31:40 raidserver kernel: [ 1100.010141] end_request: I/O error, dev sdg, sector 0 Mar 18 23:31:40 raidserver kernel: [ 1100.010310] sd 8:0:2:0: [sdh] Unhandled error code Mar 18 23:31:40 raidserver kernel: [ 1100.010314] sd 8:0:2:0: [sdh] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT Mar 18 23:31:40 raidserver kernel: [ 1100.010321] end_request: I/O error, dev sdh, sector 8 Mar 18 23:32:11 raidserver kernel: [ 1131.010514] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010523] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010545] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010551] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010562] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010567] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010580] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010585] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010596] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010601] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010612] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:11 raidserver kernel: [ 1131.010617] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010054] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010063] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010084] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010090] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010101] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010107] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010119] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010125] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010136] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010142] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010153] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:32:42 raidserver kernel: [ 1162.010159] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010053] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010062] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010083] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010089] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010100] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010106] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010118] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010123] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010134] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010139] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010150] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:33:13 raidserver kernel: [ 1193.010156] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:33:16 raidserver ata_id[2906]: HDIO_GET_IDENTITY failed for '/dev/sdj' Mar 18 23:33:16 raidserver ata_id[2910]: HDIO_GET_IDENTITY failed for '/dev/sdk' Mar 18 23:33:16 raidserver ata_id[2911]: HDIO_GET_IDENTITY failed for '/dev/sdi' Mar 18 23:33:34 raidserver kernel: [ 1213.857434] md: md0 still in use. Mar 18 23:33:40 raidserver kernel: [ 1219.907801] EXT4-fs: mballoc: 0 blocks 0 reqs (0 success) Mar 18 23:33:40 raidserver kernel: [ 1219.907801] EXT4-fs: mballoc: 0 extents scanned, 0 goal hits, 0 2^N hits, 0 breaks, 0 lost Mar 18 23:33:40 raidserver kernel: [ 1219.907801] EXT4-fs: mballoc: 0 generated and it took 0 Mar 18 23:33:40 raidserver kernel: [ 1219.907801] EXT4-fs: mballoc: 0 preallocated, 0 discarded Mar 18 23:34:10 raidserver kernel: [ 1250.010051] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010061] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010087] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010093] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010109] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010115] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010130] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010135] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010150] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010155] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010166] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:10 raidserver kernel: [ 1250.010171] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010517] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010528] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010550] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010556] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010572] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010578] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010592] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010597] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010611] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010617] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010627] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 Mar 18 23:34:41 raidserver kernel: [ 1281.010633] /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 Mar 18 23:35:06 raidserver kernel: Kernel logging (proc) stopped. Mar 18 23:35:06 raidserver rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="1027" x-info="http://www.rsyslog.com"] exiting on signal 15. Mar 18 23:39:31 raidserver kernel: imklog 4.2.0, log source = /var/run/rsyslog/kmsg started. Mar 18 23:39:31 raidserver rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="647" x-info="http://www.rsyslog.com"] (re)start Mar 18 23:39:31 raidserver rsyslogd: rsyslogd's groupid changed to 102 Mar 18 23:39:31 raidserver rsyslogd: rsyslogd's userid changed to 101 -- Stephan E Stachurski 773-315-1684 ses1984@gmail.com -- Stephan E Stachurski 773-315-1684 ses1984@gmail.com -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html-- Stephan E Stachurski 773-315-1684 ses1984@gmail.com -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.htmlAs Niel Brown stated in his reply, the issue is more complicated than that. Your array has entered a less than known state; likely while working with the critical section. Some parts may be stored as before, while others may be stored as things should look now. This is why the critical section backup and possible recovery is very important. Please follow Niel Brown's directions /very/ carefully. You probably also want to save the start of each device. dd if=/dev/container of=someplace/array_dev_X.raw bs=1024k count=64 for each device in the array (saving to separate files) would probably be a tolerable safety net to start with. You could also load these much smaller segments in hex-editors more easily.
-- Stephan E Stachurski 773-315-1684 ses1984@gmail.com -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html