On 06/07/2013 12:15 AM, Nic Wolfe wrote:

My original post had an error - sda is my boot drive, it's not part of
the array. The 6th drive is as follows:

/dev/sdg:

[trim /]

      Number   Major   Minor   RaidDevice State
this     5      65        0        5      active sync

Before you move drives around, please spend some time correlating drive
serial numbers to the "RaidDevice" number for each.  Just in case.  You
may be able to get everything you need from /dev/disk/by-id/, but with a
weird driver, I not sure.

quoted

Your reshape is barely started.  Presumably you specified a --backup
clause in the original --grow command.  You will need that file.

Embarrassingly I have no backup file from the grow operation.

mdadm should have halted if it knew it was needed.  You may be ok.

quoted

Modern mdadm should be able to force assemble this and continue without
problems.  Rather than operate within a questionable environment, I
would strongly encourage you to perform the forced assembly with a
recent live cd.  I personally use "SystemRescueCD", and I know it has
the appropriate kernel support and tools.

But.  You need to share more information about your hardware problems.
Dmesg, etc.  There are commonly-encountered configuration problems that
appear to be mysterious drive failures.  If you know all about error
recovery control, please elaborate.  Otherwise, please share the output
of "smartctl -x /dev/sdX" for all of your member devices.

My drives are connected to the machine through a poorly supported old
RAID card (rr2522) which required me to build the driver into my
kernel, so I don't think a live cd will work. I don't have enough SATA
slots in the machine to connect them all without it.

Hmmm.  That's ugly.

If it isn't obvious by now, I definitely don't know anything about
error recovery control.

Well, search the archive for terms like "scterc", "ure", and/or
"timeout".  You'll need to address the issue if you don't have
enterprise drives.

Since it's a RAID card and not just a hba each drive is presented to
the OS as a single drive JBOD array which means the OS doesn't see any
SMART info. I can see (at least some) SMART information through my
RAID card admin console and it claims they're all fine.

Drives with pending unrecoverable read errors won't make it through a
rebuild, but show "PASSED" to smart.  The details matter.

I am not having hardware problems at the moment, I only encountered
them when I had 16 drives running through the RAID card (I have
another 10 drive array). With the other array disconnected the card
seems to be behaving - there's nothing suspicious that I can see in
dmesg.

That sounds like insufficient power.

In the meantime I will see if I can put together a machine with 6 SATA
ports and attempt to hook the drives up directly rather than through
the RAID card so I can use a live CD and get the SMART information for
you.

Good plan, but do get the serial numbers vs. array role info.  Probably
should finish the reshape in such a machine, too.

Phil