Re: Raid10 reshape bug

So it turns out all you have to do to trigger a bug is:

mdadm --create -l raid10 -n 2 /dev/md1 /dev/loop0 missing
mdadm -G /dev/md1 -p o2

After changing the layout to offset, attemting to mkfs.ext4 on the raid
device results in errors and this in dmesg:

[1467312.410811] md: md1: reshape done.
[1467386.790079] handle_bad_sector: 446 callbacks suppressed
[1467386.790083] attempt to access beyond end of device
                 dm-3: rw=0, want=2127992, limit=2097152
[1467386.790096] attempt to access beyond end of device
                 dm-3: rw=0, want=2127992, limit=2097152
[1467386.790099] buffer_io_error: 2062 callbacks suppressed
[1467386.790101] Buffer I/O error on dev md1, logical block 4238, async page read
[1467386.793270] attempt to access beyond end of device
                 dm-3: rw=0, want=2127992, limit=2097152
[1467386.793277] Buffer I/O error on dev md1, logical block 4238, async page read
[1467394.422528] attempt to access beyond end of device
                 dm-3: rw=0, want=4187016, limit=2097152
[1467394.422541] attempt to access beyond end of device
                 dm-3: rw=0, want=4187016, limit=2097152
[1467394.422545] Buffer I/O error on dev md1, logical block 261616, async page read

/dev/md1:
           Version : 1.2
     Creation Time : Mon Mar  8 11:21:23 2021
        Raid Level : raid10
        Array Size : 1046528 (1022.00 MiB 1071.64 MB)
     Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
      Raid Devices : 2
     Total Devices : 1
       Persistence : Superblock is persistent

       Update Time : Mon Mar  8 11:24:10 2021
             State : clean, degraded
    Active Devices : 1
   Working Devices : 1
    Failed Devices : 0
     Spare Devices : 0

            Layout : offset=2
        Chunk Size : 512K

Consistency Policy : resync

              Name : hyper1:1  (local to host hyper1)
              UUID : 69618fc3:c6abd8de:8458d647:1c242e1a
            Events : 3409

    Number   Major   Minor   RaidDevice State
       0     253        3        0      active sync   /dev/dm-3
       -       0        0        1      removed

/dev/hyper1/leg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 69618fc3:c6abd8de:8458d647:1c242e1a
           Name : hyper1:1  (local to host hyper1)
  Creation Time : Mon Mar  8 11:21:23 2021
     Raid Level : raid10
   Raid Devices : 2

 Avail Dev Size : 2095104 (1023.00 MiB 1072.69 MB)
     Array Size : 1046528 (1022.00 MiB 1071.64 MB)
  Used Dev Size : 2093056 (1022.00 MiB 1071.64 MB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=2048 sectors
          State : clean
    Device UUID : 476f8e72:76084630:c33c16e4:7c987659

    Update Time : Mon Mar  8 11:24:10 2021
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : e5995957 - correct
         Events : 3409

         Layout : offset=2
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : A. ('A' == active, '.' == missing, 'R' == replacing)


Phillip Susi writes:

In the process of upgrading a xen server I broke the previous raid1 and
used the removed disk to create a new raid10 to prepare the new install.
I think initially I created it in the default near configuration, so I
reshaped it to offset with 1M chunk size.  I got the domUs up and
running again and was pretty happy with the result, so I blew away the
old system disk and added that disk to the new array and allowed it to
sync.  Then I thought that the 1M chunk size was hurting performance, so
I requested a reshape to a 256k chunk size with mdadm -G /dev/md0 -c
256.  It looked like it was proceeding fine so I went home for the
night.

When I came in this morming, mdadm -D showed that the reshape was
complete, but I started getting ELF errors and such running various
programs and I started to get a feeling that something had gone horribly
wrong.  At one point I was trying to run blockdev --getsz and isntead
the system somehow ran findmnt.  mdadm -E showed that there was a very
large unused section of the disk both before and after.  This is
probably because I had used -s to restrict the used size of the device
to be only 256g instead of the full 2tb so it wouldn't take so long to
resync, and since there was plenty of unused space, md decided to just
write back the new layout stripes in unused space further down the disk.
At this point I rebooted and grub could not recognize the filesystem.  I
booted other media and tried an e2fsck but it had so many complaints,
one of which being that the root directory was not, in fact, a directory
so it deleted it that I just gave up and started reinstalling and
restoring the domU from backup.

Clearly somehow the reshape process did NOT write the data back to the
disk in the correct place.  This was using debian testing with linux
5.10.0 and mdadm v4.1.

I will try to reproduce it in a vm at some point.