On Fri, 27 Nov 2009, Asdo wrote:

Asdo wrote:

quoted

....
Without knowing this I will probably opt for your way: rsync data out, 
starting from the smallest and most important stuff...
...

I had another thought:

If I take the computer offline, boot with a livecd (one of the worst messes 
here is that the root filesystem is also in that array), run the raid6 array 
in READONLY MODE and maybe without spares, then I start a check (scrub) ...

If drives are kicked I can probably reasseble --force the array and it's like 
nothing happened, right?

If *more* than 2 drives fail you would need to --force.  Also, when you do that
(I have done it before), you need to fsck the filesystem and often many of
your files will end up in /lost+found, depending on how bad it is.  But in all
of my tests, the FS was R/W, not R/O, so I am unsure of the outcome, it sounds
like a possibility.  With SW/RAID-6 I have lost two disks before and suffered
no problems at all (you can't use Western Digital Velociraptors in RAID), I was
able to copy the data that I needed off of the array without any issues but my
server was not being heavily utilized as yours is.

I think you need to run a smartctl on each of the drives, e.g.:

for disk in /dev/sd[a-z]
do
   echo "disk $disk" >> /tmp/a
   smartctl -a $disk >> /tmp/a
done

Inspect each disk, is there really a failing disk?..

Since it was mounted readonly I think it would be clean...

Only problem would be if 1 or more drives definitively die during the 
procedure, but I hope this is unlikely...
If less than 3 drives die I can still reassemble --force, and take the data 
out (at least SOME data, then if it degrades, reassemble again and try to get 
out data from another location...)

Do you agree?

I agree, but would you not want to just rsync the data off first before going
through all of this?

I am starting to think that during the procedure for taking the data out 
and/or attempt first scrubbing the main problem are write accesses to the 
array, because if rebuild starts on a spare and then fails again and then 
there were writes in the middle... I think I end up doomed. Probably even 
reassemble --force would refuse to work on me. What do you think?

I think you have a point there..  In my opinion:

1. Check each disk, how bad is it, really? (it seems like your array and
    disks are fine, one disk may have a re-allocated sector or two,
    nothing to worry about) in all seriousness.  Do any of the attributes say
    *FAILING NOW*?

2. If everything looks OK, copy all of the data off while the system is ONLINE
    and WORKING, it will be *MUCH* more difficult trying to extract pieces of
    data using dd_rescue and friends vs. just rsyncing the entire array to
    another host.

3. Do you not have another host to rsync to?  If that is the case then we may
    need to approach this problem from a different angle.  E.g., making it
    read-only after booting from a LiveCD may not be a bad idea, but doing that
    BEFORE you rsync'd all the data off is still risking all of the data on the
    array, whereas since it is currently up and running you could at least make
    a point-in-time copy of all of the data that lives there right now.

Justin.