Hi Phil,

On 11/05/2020 15:18, Phil Turmel wrote:

quoted

So I've come to the conclusion that the only way forward is to use 
`mdadm --create` and hope I get the array back that way, with new 
superblocks.

Yes.  Be sure to always include --assume-clean in these trials.

quoted

However, it's my understanding that you need to add these disks in the 
correct order - and given I have 7 disks, that's 5040 possible 
permutations! The original four disks show their device roles, so I'm 
/assuming/ that's the order in which they need adding:

Yes, existing superblocks can be trusted.  You should show the complete 
"mdadm -E" output for each of these member devices for our reference.

My complete mdadm -E output:

sosh@toothless:/$ sudo mdadm -E /dev/sda /dev/sdb /dev/sdc /dev/sdd 
/dev/sdf /dev/sdg /dev/sdh
/dev/sda:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x1
      Array UUID : f3667bf3:e2b718c2:22cbea68:428ea6ca
            Name : toothless:WDRAID
   Creation Time : Sat Oct 22 10:52:47 2016
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650675200 (13971.97 GiB 15002.29 GB)
   Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262064 sectors, after=944 sectors
           State : clean
     Device UUID : 08c24be4:d352cbb0:edd50ba7:3a70e02e

Internal Bitmap : 8 sectors from superblock
     Update Time : Sat May  2 09:49:27 2020
   Bad Block Log : 512 entries available at offset 24 sectors
        Checksum : b1df527c - correct
          Events : 2687251

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : Active device 0
    Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x1
      Array UUID : f3667bf3:e2b718c2:22cbea68:428ea6ca
            Name : toothless:WDRAID
   Creation Time : Sat Oct 22 10:52:47 2016
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650675200 (13971.97 GiB 15002.29 GB)
   Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=944 sectors
           State : clean
     Device UUID : 80d493f1:9649882b:bdcdeceb:cb61f937

Internal Bitmap : 8 sectors from superblock
     Update Time : Sat May  2 09:49:27 2020
   Bad Block Log : 512 entries available at offset 72 sectors
        Checksum : 87de4546 - correct
          Events : 2687251

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : Active device 2
    Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x1
      Array UUID : f3667bf3:e2b718c2:22cbea68:428ea6ca
            Name : toothless:WDRAID
   Creation Time : Sat Oct 22 10:52:47 2016
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650675200 (13971.97 GiB 15002.29 GB)
   Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=944 sectors
           State : clean
     Device UUID : 078aa94c:de9ad5e8:dcb5b43c:bf12292f

Internal Bitmap : 8 sectors from superblock
     Update Time : Sat May  2 09:49:27 2020
   Bad Block Log : 512 entries available at offset 72 sectors
        Checksum : e838e526 - correct
          Events : 2687251

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : Active device 1
    Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x1
      Array UUID : f3667bf3:e2b718c2:22cbea68:428ea6ca
            Name : toothless:WDRAID
   Creation Time : Sat Oct 22 10:52:47 2016
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650675200 (13971.97 GiB 15002.29 GB)
   Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262056 sectors, after=944 sectors
           State : clean
     Device UUID : 47eb651d:9d5eb96b:3b8e9d78:882f5f49

Internal Bitmap : 8 sectors from superblock
     Update Time : Sat May  2 09:49:27 2020
   Bad Block Log : 512 entries available at offset 72 sectors
        Checksum : 91f7ff4f - correct
          Events : 2687251

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : Active device 3
    Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x1
      Array UUID : f3667bf3:e2b718c2:22cbea68:428ea6ca
            Name : toothless:WDRAID
   Creation Time : Sat Oct 22 10:52:47 2016
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650675200 (13971.97 GiB 15002.29 GB)
   Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262064 sectors, after=944 sectors
           State : clean
     Device UUID : 00000000:00000000:00000000:00000000

Internal Bitmap : 8 sectors from superblock
     Update Time : Sat May  2 09:49:27 2020
   Bad Block Log : 512 entries available at offset 24 sectors
        Checksum : b1df527c - expected 46dbf779
          Events : 2687251

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : spare
    Array State : ....... ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : f3667bf3:e2b718c2:22cbea68:428ea6ca
            Name : toothless:WDRAID
   Creation Time : Sat Oct 22 10:52:47 2016
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650675200 (13971.97 GiB 15002.29 GB)
   Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262064 sectors, after=944 sectors
           State : active
     Device UUID : 00000000:00000000:00000000:00000000

     Update Time : Thu Jan  1 01:00:00 1970
        Checksum : 0 - expected e7edc1ab
          Events : 0

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : spare
    Array State : ....... ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : f3667bf3:e2b718c2:22cbea68:428ea6ca
            Name : toothless:WDRAID
   Creation Time : Sat Oct 22 10:52:47 2016
      Raid Level : raid6
    Raid Devices : 7

  Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
      Array Size : 14650675200 (13971.97 GiB 15002.29 GB)
   Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262064 sectors, after=944 sectors
           State : active
     Device UUID : 00000000:00000000:00000000:00000000

     Update Time : Thu Jan  1 01:00:00 1970
        Checksum : 0 - expected e7edc1ab
          Events : 0

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : spare
    Array State : ....... ('A' == active, '.' == missing, 'R' == replacing)

quoted

So I've tried all six permutations of the devices showing as "spare" 
at the end and I can never get a sensible filesystem out when I do a 
--create.

Does anyone have any other ideas, or can offer some wisdom into what 
to do next? Otherwise I'm writing a shell script to test all 5040 
permutations...

It isn't just order that matters.   You must get the right data offset 
and chunk size.  Defaults have changed over the years, and offsets 
typically change (+/- 1 chunk) during reshapes.

You'll probably have to manually specify this stuff.  Be sure to use the 
latest released version of mdadm, even if you have to compile it yourself.

If your data offsets are at least a couple megabytes, consider 
partitioning these disks at the same time as you reconstruct--simply 
adjust the data offset for the start sector of the partition.  This will 
avoid future issues with stupid mobos.  (You aren't the first to suffer 
from this.)

So I've now tried doing this and sadly haven't really gotten anywhere. 
Given the output of mdadm -E, I've specified the chunk size as 512K, and 
the data offset as 134MB (given the reported offset of 262144 sectors * 
sector size of 512 bytes on the devices).

Given your statement that I could trust the existing superblocks, I 
haven't messed around with the order of those and instead I've just been 
messing around with the order of the three disks which are unhappy. So 
I've been putting sda first (active 0), then sdc (active 1), sdb (active 
2), and sdd (active 3). As I said last time, I'm using overlay images to 
avoid screwing up the disks. My create command line is as below:

mdadm --create /dev/md1 --chunk=512 --level=6 --assume-clean 
--data-offset=134M --raid-devices=7 /dev/mapper/sda /dev/mapper/sdc 
/dev/mapper/sdb /dev/mapper/sdd ${UNHAPPY_DISK1} ${UNHAPPY_DISK2} 
${UNHAPPY_DISK3}

Apologies that it's taken a while to respond to your mail, I just 
haven't had much time to look at this during the week.


Best Regards,
Sam