Hello,

I recently migrated from btrfs raid-5 to md raid-5 and because I didn't 
have enough drives to go directly md raid-5, I basicly did this:
- 2 disks raid-0
- added 3rd disk with grow when I had moved data, still raid-0
- added 4th disk with grow when I had moved more data, still raid-0
- added 5th disk with grow and converted to raid-5

Everything went fine other than I could not add internal bitmap because 
array was created as raid-0 originally but thats not an issue, I'll 
store bitmap to my /boot, it's raid-1 on SSD and ext4.

Today I noticed that my array parity layout is parity-last and I started 
to wonder why is that, then I did remember that there was some strange 
things happening when I did add 4th drive and I made testing and was 
able to replicate what happened.

[root@nas ~]# mdadm -C /dev/md/testraid -l 0 -n 2 -N testraid 
/dev/loop21 /dev/loop22
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/testraid started.


[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
            Version : 1.2
      Creation Time : Fri Dec 10 22:52:01 2021
         Raid Level : raid0
         Array Size : 98304 (96.00 MiB 100.66 MB)
       Raid Devices : 2
      Total Devices : 2
        Persistence : Superblock is persistent

        Update Time : Fri Dec 10 22:52:01 2021
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0

             Layout : -unknown-
         Chunk Size : 512K

Consistency Policy : none

               Name : nas:testraid  (local to host nas)
               UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
             Events : 0

     Number   Major   Minor   RaidDevice State
        0       7       21        0      active sync   /dev/loop21
        1       7       22        1      active sync   /dev/loop22


[root@nas ~]# mdadm --grow /dev/md/testraid -n 3 -l 0 --add /dev/loop23
mdadm: level of /dev/md/testraid changed to raid4
mdadm: added /dev/loop23


[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
            Version : 1.2
      Creation Time : Fri Dec 10 22:52:01 2021
         Raid Level : raid0
         Array Size : 147456 (144.00 MiB 150.99 MB)
       Raid Devices : 3
      Total Devices : 3
        Persistence : Superblock is persistent

        Update Time : Fri Dec 10 22:53:33 2021
              State : clean
     Active Devices : 3
    Working Devices : 3
     Failed Devices : 0
      Spare Devices : 0

         Chunk Size : 512K

Consistency Policy : none

               Name : nas:testraid  (local to host nas)
               UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
             Events : 18

     Number   Major   Minor   RaidDevice State
        0       7       21        0      active sync   /dev/loop21
        1       7       22        1      active sync   /dev/loop22
        3       7       23        2      active sync   /dev/loop23


[root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0 --add /dev/loop24
mdadm: level of /dev/md/testraid changed to raid4
mdadm: added /dev/loop24
mdadm: Need to backup 6144K of critical section..


[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
            Version : 1.2
      Creation Time : Fri Dec 10 22:52:01 2021
         Raid Level : raid4
         Array Size : 196608 (192.00 MiB 201.33 MB)
      Used Dev Size : 49152 (48.00 MiB 50.33 MB)
       Raid Devices : 5
      Total Devices : 4
        Persistence : Superblock is persistent

        Update Time : Fri Dec 10 22:59:56 2021
              State : clean, degraded
     Active Devices : 4
    Working Devices : 4
     Failed Devices : 0
      Spare Devices : 0

         Chunk Size : 512K

Consistency Policy : resync

               Name : nas:testraid  (local to host nas)
               UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
             Events : 39

     Number   Major   Minor   RaidDevice State
        0       7       21        0      active sync   /dev/loop21
        1       7       22        1      active sync   /dev/loop22
        3       7       23        2      active sync   /dev/loop23
        4       7       24        3      active sync   /dev/loop24
        -       0        0        4      removed


[root@nas ~]# mdadm --grow /dev/md/testraid -n 5 -l 5 --add /dev/loop25
mdadm: level of /dev/md/testraid changed to raid5
mdadm: added /dev/loop25


[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
            Version : 1.2
      Creation Time : Fri Dec 10 22:52:01 2021
         Raid Level : raid5
         Array Size : 196608 (192.00 MiB 201.33 MB)
      Used Dev Size : 49152 (48.00 MiB 50.33 MB)
       Raid Devices : 5
      Total Devices : 5
        Persistence : Superblock is persistent

        Update Time : Fri Dec 10 23:00:39 2021
              State : clean
     Active Devices : 5
    Working Devices : 5
     Failed Devices : 0
      Spare Devices : 0

             Layout : parity-last
         Chunk Size : 512K

Consistency Policy : resync

               Name : nas:testraid  (local to host nas)
               UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
             Events : 59

     Number   Major   Minor   RaidDevice State
        0       7       21        0      active sync   /dev/loop21
        1       7       22        1      active sync   /dev/loop22
        3       7       23        2      active sync   /dev/loop23
        4       7       24        3      active sync   /dev/loop24
        5       7       25        4      active sync   /dev/loop25


Notice how array stay at raid4 degraded after I add 4th drive. Why is 
this happening? It doesn't happen when I added 3rd drive, it did reshape 
it through raid-4 but when it was finished it returned raid-0.

I also did another test where I made extra step after adding 4th disk:

[root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0
mdadm: level of /dev/md/testraid changed to raid0
[root@nas ~]# cat /proc/mdstat
Personalities : [raid1] [raid0] [linear] [raid6] [raid5] [raid4]
md118 : active raid0 loop24[4] loop23[3] loop22[1] loop21[0]
       196608 blocks super 1.2 512k chunks


And when I did add 5th drive and converted array to raid-5, it correctly 
defaults layout to left-symmetric.

Is this some sort of bug or working as intented? Also why is this "Need 
to backup 6144K of critical section.." happening when I add 4th disk but 
not when I add 3rd disk?


distro: Fedora 35
kernel:  5.15.6
mdadm:  v4.2-rc2


// JiiPee