On Fri, Mar 12, 2010 at 12:17 AM, Michael Evans [off-list ref] wrote:

On Thu, Mar 11, 2010 at 11:51 PM, Jonathan Gordon
[off-list ref] wrote:

quoted

Upon reboot, my machine began recovering from a raid1 failure.
Querying mdadm yielded the following:

jgordon@kubuntu:~$ sudo mdadm --detail /dev/md0
[sudo] password for jgordon:
/dev/md0:
      Version : 00.90
 Creation Time : Mon Sep 11 06:35:17 2006
   Raid Level : raid1
   Array Size : 242187776 (230.97 GiB 248.00 GB)
 Used Dev Size : 242187776 (230.97 GiB 248.00 GB)
 Raid Devices : 2
 Total Devices : 2
Preferred Minor : 0
  Persistence : Superblock is persistent

  Update Time : Thu Mar 11 18:09:25 2010
        State : clean, degraded, recovering
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
 Spare Devices : 1

 Rebuild Status : 26% complete

         UUID : 7fd22081:c39cb3e4:21109eec:10ecdf10
       Events : 0.5260272

  Number   Major   Minor   RaidDevice State
     2       8        1        0      spare rebuilding   /dev/sda1
     1       8       17        1      active sync   /dev/sdb1

After some time, the rebuild seemed to complete, but the State seemed
to switch alternately between "active, degraded" and "clean,
degraded". Addiontally, the state for /dev/sda1 seems to continue to
stay in "spare rebuilding". This is the current output:

jgordon@kubuntu:~$ sudo mdadm -D /dev/md0
[sudo] password for jgordon:
/dev/md0:
      Version : 00.90
 Creation Time : Mon Sep 11 06:35:17 2006
   Raid Level : raid1
   Array Size : 242187776 (230.97 GiB 248.00 GB)
 Used Dev Size : 242187776 (230.97 GiB 248.00 GB)
 Raid Devices : 2
 Total Devices : 2
Preferred Minor : 0
  Persistence : Superblock is persistent

  Update Time : Thu Mar 11 23:07:59 2010
        State : clean, degraded
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
 Spare Devices : 1

         UUID : 7fd22081:c39cb3e4:21109eec:10ecdf10
       Events : 0.5273340

  Number   Major   Minor   RaidDevice State
     2       8        1        0      spare rebuilding   /dev/sda1
     1       8       17        1      active sync   /dev/sdb1

Additionally, /var/log/kern.log is getting filled with the following:

Mar 11 19:19:14 jigme kernel: [ 6596.236366] ata4: EH complete
Mar 11 19:19:16 jigme kernel: [ 6598.104676] ata4.00: exception Emask
0x0 SAct 0x0 SErr 0x0 action 0x0
Mar 11 19:19:16 jigme kernel: [ 6598.104683] ata4.00: BMDMA stat 0x24
Mar 11 19:19:16 jigme kernel: [ 6598.104692] ata4.00: cmd
25/00:08:ff:b0:e0/00:00:15:00:00/e0 tag 0 dma 4096 in
Mar 11 19:19:16 jigme kernel: [ 6598.104694]          res
51/40:00:04:b1:e0/40:00:15:00:00/e0 Emask 0x9 (media error)
Mar 11 19:19:16 jigme kernel: [ 6598.104698] ata4.00: status: { DRDY ERR }
Mar 11 19:19:16 jigme kernel: [ 6598.104702] ata4.00: error: { UNC }
Mar 11 19:19:16 jigme kernel: [ 6598.120352] ata4.00: configured for UDMA/133
Mar 11 19:19:16 jigme kernel: [ 6598.120371] sd 3:0:0:0: [sdb]
Unhandled sense code
Mar 11 19:19:16 jigme kernel: [ 6598.120375] sd 3:0:0:0: [sdb] Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar 11 19:19:16 jigme kernel: [ 6598.120380] sd 3:0:0:0: [sdb] Sense
Key : Medium Error [current] [descriptor]
Mar 11 19:19:16 jigme kernel: [ 6598.120388] Descriptor sense data
with sense descriptors (in hex):
Mar 11 19:19:16 jigme kernel: [ 6598.120392]         72 03 11 04 00 00
00 0c 00 0a 80 00 00 00 00 00
Mar 11 19:19:16 jigme kernel: [ 6598.120412]         15 e0 b1 04
Mar 11 19:19:16 jigme kernel: [ 6598.120420] sd 3:0:0:0: [sdb] Add.
Sense: Unrecovered read error - auto reallocate failed
Mar 11 19:19:16 jigme kernel: [ 6598.120428] end_request: I/O error,
dev sdb, sector 367046916
Mar 11 19:19:16 jigme kernel: [ 6598.120446] ata4: EH complete
Mar 11 19:19:16 jigme kernel: [ 6598.120744] raid1: sdb: unrecoverable
I/O read error for block 367046784
Mar 11 19:19:17 jigme kernel: [ 6599.164052] md: md0: recovery done.
Mar 11 19:19:17 jigme kernel: [ 6599.460124] RAID1 conf printout:
Mar 11 19:19:17 jigme kernel: [ 6599.460145]  --- wd:1 rd:2
Mar 11 19:19:17 jigme kernel: [ 6599.460160]  disk 0, wo:1, o:1, dev:sda1
Mar 11 19:19:17 jigme kernel: [ 6599.460170]  disk 1, wo:0, o:1, dev:sdb1
Mar 11 19:19:17 jigme kernel: [ 6599.460178] RAID1 conf printout:
Mar 11 19:19:17 jigme kernel: [ 6599.460185]  --- wd:1 rd:2
Mar 11 19:19:17 jigme kernel: [ 6599.460195]  disk 0, wo:1, o:1, dev:sda1
Mar 11 19:19:17 jigme kernel: [ 6599.460204]  disk 1, wo:0, o:1, dev:sdb1
Mar 11 19:19:22 jigme kernel: [ 6604.165111] RAID1 conf printout:
Mar 11 19:19:22 jigme kernel: [ 6604.165117]  --- wd:1 rd:2
Mar 11 19:19:22 jigme kernel: [ 6604.165122]  disk 0, wo:1, o:1, dev:sda1
Mar 11 19:19:22 jigme kernel: [ 6604.165125]  disk 1, wo:0, o:1, dev:sdb1
Mar 11 19:19:22 jigme kernel: [ 6604.165128] RAID1 conf printout:
Mar 11 19:19:22 jigme kernel: [ 6604.165131]  --- wd:1 rd:2
Mar 11 19:19:22 jigme kernel: [ 6604.165134]  disk 0, wo:1, o:1, dev:sda1
Mar 11 19:19:22 jigme kernel: [ 6604.165137]  disk 1, wo:0, o:1, dev:sdb1
...
Mar 11 23:16:28 jigme kernel: [20830.889380] RAID1 conf printout:
Mar 11 23:16:28 jigme kernel: [20830.889386]  --- wd:1 rd:2
Mar 11 23:16:28 jigme kernel: [20830.889391]  disk 0, wo:1, o:1, dev:sda1
Mar 11 23:16:28 jigme kernel: [20830.889394]  disk 1, wo:0, o:1, dev:sdb1
Mar 11 23:16:28 jigme kernel: [20830.889397] RAID1 conf printout:
Mar 11 23:16:28 jigme kernel: [20830.889399]  --- wd:1 rd:2
Mar 11 23:16:28 jigme kernel: [20830.889403]  disk 0, wo:1, o:1, dev:sda1
Mar 11 23:16:28 jigme kernel: [20830.889406]  disk 1, wo:0, o:1, dev:sdb1

The "RAID1 conf printout:" messages appear every few seconds or so.

Machine info:

jgordon@kubuntu:~$ uname -a
Linux kubuntu 2.6.31-20-386 #57-Ubuntu SMP Mon Feb 8 11:42:49 UTC 2010
i686 GNU/Linux

Any idea what I can do to resolve this?

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Replace your failing disk; from the look of the kernel log and the
description of the issue I'd say your drive is out of spare sectors
and would fail a S.M.A.R.T. test.

If you require more proof start reading up on how to use the smartctl
command from the smartmontools package (may have dashes/etc in your
package manager).

http://sourceforge.net/apps/trac/smartmontools/wiki/TocDoc

Reading more carefully, I notice you're in a most unfortunate
situation.  It's failing while trying to READ your 'ACTIVE' disc.

You should buy two NEW drives, attempt to copy the current active
member of the array to a NEW drive using something like
http://www.gnu.org/software/ddrescue/ddrescue.html

For anything that won't copy you /MIGHT/ try the other member of the
array if it used to be in place.  Chances are good that stale data
from a section MAY be better than no data at all.  You MUST perform an
FSCK prior to mounting the newly re-created array; you should try to
determine which files occupy any 'recovered' sectors and validate that
they are intact or replace them from other duplicates.

How to do that depends very much on what filesystem you have on the array.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html