Re: array went wonky
From: Gimpbully <hidden>
Date: 2013-05-03 06:47:43
Anyone? I'm so close I can taste it. It looks like the superblocks on the other disks think that sdf1 is still defined as a spare, but it's a ddrescue of what used to be /dev/sdc. Is there a way to update the superblocks? On May 2, 2013, at 7:29 AM, Gimpbully [off-list ref] wrote:
Sorry, I realize I had md127 started. So I stopped it and did the assemble and got the following (the problem is sdf isn't supposed to be a spare at all, it's supposed to be a ddrescue'd sdc: eduardo ~ # mdadm --stop /dev/md127 mdadm: stopped /dev/md127 eduardo ~ # mdadm --assemble --force /dev/md127 /dev/sd{f,g,e,a}1 mdadm: /dev/md127 assembled from 3 drives and 1 spare - not enough to start the array. eduardo ~ # eduardo ~ # mdadm --examine /dev/sd{f,g,e,a}1 | egrep 'Event|/dev/sd' /dev/sdf1: Events : 25979 this 5 8 81 5 spare /dev/sdf1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sdg1: Events : 25979 this 1 8 17 1 active sync /dev/sdb1 0 0 8 33 0 active sync /dev/sdc1 1 1 8 17 1 active sync /dev/sdb1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sde1: Events : 25979 this 2 8 65 2 active sync /dev/sde1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sda1: Events : 25979 this 4 8 1 4 active sync /dev/sda1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 On May 2, 2013, at 7:18 AM, Gimpbully [off-list ref] wrote:quoted
This is very odd. When trying to force an assemble, it's syaing no superblock on /dev/sdf1, but the --examine output appears to say otherwise… Does anyone have any idea how to do this without doing a re-creation? eduardo ~ # mdadm --examine /dev/sd{f,g,e,a}1 | egrep 'Event|/dev/sd' /dev/sdf1: Events : 25979 this 5 8 81 5 spare /dev/sdf1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sdg1: Events : 25971 this 1 8 17 1 active sync /dev/sdb1 0 0 8 33 0 active sync /dev/sdc1 1 1 8 17 1 active sync /dev/sdb1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sde1: Events : 25979 this 2 8 65 2 active sync /dev/sde1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sda1: Events : 25979 this 4 8 1 4 active sync /dev/sda1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 eduardo ~ # On May 1, 2013, at 11:59 AM, Gimpbully [off-list ref] wrote:quoted
Alright, so I've ddrescued 2 disks now: sda - clean sdb - SMART errors, kicked out of array sdc - SMART errors, kicked out of array sde - ddrescued copy of sdb sdf - original failed disk, replaced. is now a ddrescued copy of sdc So I run eduardo ~ # mdadm --assemble --force /dev/md127 /dev/sd{f,g,e,a}1 mdadm: cannot open device /dev/sdf1: Device or resource busy mdadm: /dev/sdf1 has no superblock - assembly aborted eduardo ~ # Any ideas? On Apr 30, 2013, at 7:48 AM, Gimpbully [off-list ref] wrote:quoted
I have no intentions of recreating at all. I fully understand the implications (I've had stripe orders go bad on truly massive scales before, I don't ever wanna experience that again), I just don't know mdadm as well as other vendor storage. I was able to ddrescue a copy of sdb with no errors. I assembled the array and got a file system metadata dump (this is HUGE) and it's currently rebuilding before I even look at data. Thank you all for your help, patience was absolutely key here. On Apr 30, 2013, at 7:45 AM, Phil Turmel [off-list ref] wrote:quoted
On 04/30/2013 02:20 AM, Sam Bingner wrote:quoted
On Apr 29, 2013, at 4:33 PM, "Gimpbully" [off-list ref] wrote:quoted
On Apr 13, 2013, at 7:20 AM, Sam Bingner [off-list ref] wrote:quoted
After that you can try to recreate the array with the proper order (sdc1, sdb1, sde1, missing, sda1) and copy data off or add the spare in again depending on if you were able to recover all the data wih GNU ddrescue.What do you mean recreate? what's the specific command? something like: mdadm --create --assume-clean --level=5 --raid-devices=5 /dev/md127 /dev/sdc1 /dev/sdb1 /sdv/sde1 missing /dev/sda1Don't recreate it - I said the wrong thing... You want to do an assemble on them with force if possible... Recreate is last ditch and make sure you have another copy if you do the previous command in case it doesn't work right due to offsets etc... try: mdadm --stop /dev/md127 mdadm --assemble --force /dev/md127 /dev/sd{c,b,e,a}1Yes.quoted
If you DO need to recreate it, what you showed looks correct.NO! The OP has *not* shared sufficient information on the array members to say that. Since it has "worked for years", the odds of an offset error is *very* high. Chunk size defaults are also likely to be different. *Complete* output of "mdadm -E" for the array members is needed before any "--create" operation is attempted. Plus the distro info, kernel version, and mdadm version . Phil
-- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html