Hello,
until recently I was using Unraid with 8 disks in an XFS array.
I installed Debian on a spare SSD on the same machine and started
migrating disk by disk from the Unraid array to a raid5 array using a
second server as temporay storage. So I switched between Debian and
Unraid a lot to copy data and remove/add drives from/to arrays.
From the beginning I always had to assemble the array without /dev/sdd,
add it afterwards and let it rebuild - since the array was working fine
afterwards I didn't really think much of it.
Appearently Unraid always overwrote the superblock of that 1 (and later
2) disks (/dev/sdc and /dev/sdd) when I switched between the two OSs and
now mdadm isn't recognizing those 2 disks and I can't assemble the array
obviously.
At least that's what I think happened, since file tells me that the
first 32k bytes are XFS:
# losetup -o 32768 -f /dev/sdd
# file -s /dev/loop0
/dev/loop0: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
Right now mdadm only assembles 5 (instead of 7) disks as spares into an
inactive array at boot:
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : inactive sde[6](S) sdf1[5](S) sdg[1](S) sdh[3](S) sdi[8](S)
21487980896 blocks super 1.2
unused devices: <none>
My system:
Linux titan 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64
GNU/Linux
mdadm - v4.1 - 2018-10-01
Maybe I could try to assemble it with assume-clean and read-only?
I found some pages in the wiki but I'm not 100% sure that they will
solve my problem and I don't want to make things worse.
https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID
https://raid.wiki.kernel.org/index.php/RAID_Recovery
I attached the remaining system information as textfiles in a zipfile,
since the outputs are pretty long - i hope thats ok.
Thanks,
best regards
Clemens
On 30/09/2021 18:21, Clemens Kiehas wrote:
Hello,
until recently I was using Unraid with 8 disks in an XFS array.
I installed Debian on a spare SSD on the same machine and started
migrating disk by disk from the Unraid array to a raid5 array using a
second server as temporay storage. So I switched between Debian and
Unraid a lot to copy data and remove/add drives from/to arrays.
From the beginning I always had to assemble the array without /dev/sdd,
add it afterwards and let it rebuild - since the array was working fine
afterwards I didn't really think much of it.
Not wise - as you've found out it was trashing your redundancy ...
Appearently Unraid always overwrote the superblock of that 1 (and later
2) disks (/dev/sdc and /dev/sdd) when I switched between the two OSs and
now mdadm isn't recognizing those 2 disks and I can't assemble the array
obviously.
At least that's what I think happened, since file tells me that the
first 32k bytes are XFS:
# losetup -o 32768 -f /dev/sdd
# file -s /dev/loop0
/dev/loop0: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
That feels good, but I'm not sure how to proceed from here.
Right now mdadm only assembles 5 (instead of 7) disks as spares into an
inactive array at boot:
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : inactive sde[6](S) sdf1[5](S) sdg[1](S) sdh[3](S) sdi[8](S)
21487980896 blocks super 1.2
unused devices: <none>
My system:
Linux titan 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64
GNU/Linux
mdadm - v4.1 - 2018-10-01
Maybe I could try to assemble it with assume-clean and read-only?
I found some pages in the wiki but I'm not 100% sure that they will
solve my problem and I don't want to make things worse.
https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID
https://raid.wiki.kernel.org/index.php/RAID_Recovery
The first page talks about overlays. THEY WILL PROTECT YOUR DATA.
Make sure every disk is overlayed, then try assembling the overlays with
your assume-clean. You won't need read-only but do it if you like it
then do.
If you can then mount and run a fsck or whatever it is over the file
system and it says everything is clean, then you can redo it without the
overlays and recover the array properly. If it's messed up, just tear
down the overlays and you're back where you started.
If it doesn't work, come back and tell us what happened (if it does
work, please let us know :-)
Cheers,
Wol
Hi Wol,
thank you.
I followed the overlay-procedure for all 7 disks and was able to start
the array with:
mdadm --create --assume-clean --level=5 --raid-devices=7 /dev/md0 $OVERLAYS
# echo $OVERLAYS
/dev/mapper/sdd /dev/mapper/sdg /dev/mapper/sdh /dev/mapper/sdf1
/dev/mapper/sde /dev/mapper/sdi /dev/mapper/sdc
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : active raid5 dm-5[6] dm-10[5] dm-8[4] dm-9[3] dm-11[2] dm-7[1] dm-6[0]
23441316864 blocks super 1.2 level 5, 512k chunk, algorithm 2
[7/7] [UUUUUUU]
bitmap: 0/30 pages [0KB], 65536KB chunk
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Sep 30 21:57:03 2021
Raid Level : raid5
Array Size : 23441316864 (22355.38 GiB 24003.91 GB)
Used Dev Size : 3906886144 (3725.90 GiB 4000.65 GB)
Raid Devices : 7
Total Devices : 7
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Thu Sep 30 21:57:03 2021
State : clean
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Name : titan:0 (local to host titan)
UUID : 45954f34:4559be76:61d18d85:f9e7ed80
Events : 0
Number Major Minor RaidDevice State
0 253 6 0 active sync /dev/dm-6
1 253 7 1 active sync /dev/dm-7
2 253 11 2 active sync /dev/dm-11
3 253 9 3 active sync /dev/dm-9
4 253 8 4 active sync /dev/dm-8
5 253 10 5 active sync /dev/dm-10
6 253 5 6 active sync /dev/dm-5
I also examined all overlay-disks and they seem to have proper superblocks:
# mdadm -E /dev/mapper/sdc
/dev/mapper/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 45954f34:4559be76:61d18d85:f9e7ed80
Name : titan:0 (local to host titan)
Creation Time : Thu Sep 30 21:57:03 2021
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813772976 (3725.90 GiB 4000.65 GB)
Array Size : 23441316864 (22355.38 GiB 24003.91 GB)
Used Dev Size : 7813772288 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=688 sectors
State : clean
Device UUID : 10d49408:d28fe8ee:58ce698e:a2d66ac0
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Sep 30 21:57:03 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 68e54840 - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
But it looks like there is no data on it?
Initially I used LVM to create /dev/md0 as a physical volume, created a
volume-group with it and a logical volume on it with all free space.
Either the physical volume doesn't exist in this state, or I don't know
to use it..
I tried pvs, pvscan, vgscan, dmsetup info, but it won't show up.
Do you think this is just a lvm problem now, or did I miss something?
Best regards,
Clemens
Am 30.09.2021 um 20:11 schrieb Wols Lists:On 30/09/2021 18:21, Clemens Kiehas wrote:
quoted
Hello,
until recently I was using Unraid with 8 disks in an XFS array.
I installed Debian on a spare SSD on the same machine and started
migrating disk by disk from the Unraid array to a raid5 array using a
second server as temporay storage. So I switched between Debian and
Unraid a lot to copy data and remove/add drives from/to arrays.
From the beginning I always had to assemble the array without
/dev/sdd, add it afterwards and let it rebuild - since the array was
working fine afterwards I didn't really think much of it.
Not wise - as you've found out it was trashing your redundancy ...
quoted
Appearently Unraid always overwrote the superblock of that 1 (and
later 2) disks (/dev/sdc and /dev/sdd) when I switched between the
two OSs and now mdadm isn't recognizing those 2 disks and I can't
assemble the array obviously.
At least that's what I think happened, since file tells me that the
first 32k bytes are XFS:
# losetup -o 32768 -f /dev/sdd
# file -s /dev/loop0
/dev/loop0: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
That feels good, but I'm not sure how to proceed from here.
quoted
Right now mdadm only assembles 5 (instead of 7) disks as spares into
an inactive array at boot:
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : inactive sde[6](S) sdf1[5](S) sdg[1](S) sdh[3](S) sdi[8](S)
21487980896 blocks super 1.2
unused devices: <none>
My system:
Linux titan 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03)
x86_64 GNU/Linux
mdadm - v4.1 - 2018-10-01
Maybe I could try to assemble it with assume-clean and read-only?
I found some pages in the wiki but I'm not 100% sure that they will
solve my problem and I don't want to make things worse.
https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID
https://raid.wiki.kernel.org/index.php/RAID_Recovery
The first page talks about overlays. THEY WILL PROTECT YOUR DATA.
Make sure every disk is overlayed, then try assembling the overlays
with your assume-clean. You won't need read-only but do it if you like
it then do.
If you can then mount and run a fsck or whatever it is over the file
system and it says everything is clean, then you can redo it without
the overlays and recover the array properly. If it's messed up, just
tear down the overlays and you're back where you started.
If it doesn't work, come back and tell us what happened (if it does
work, please let us know :-)
Cheers,
Wol
--
MatrNr: 11805432
Mobile: 06642573109
E-Mail: clemens@kiehas.at
Matrix: @clemens:matrix.spacebeard.tech