Hello,
I recently migrated from btrfs raid-5 to md raid-5 and because I didn't
have enough drives to go directly md raid-5, I basicly did this:
- 2 disks raid-0
- added 3rd disk with grow when I had moved data, still raid-0
- added 4th disk with grow when I had moved more data, still raid-0
- added 5th disk with grow and converted to raid-5
Everything went fine other than I could not add internal bitmap because
array was created as raid-0 originally but thats not an issue, I'll
store bitmap to my /boot, it's raid-1 on SSD and ext4.
Today I noticed that my array parity layout is parity-last and I started
to wonder why is that, then I did remember that there was some strange
things happening when I did add 4th drive and I made testing and was
able to replicate what happened.
[root@nas ~]# mdadm -C /dev/md/testraid -l 0 -n 2 -N testraid
/dev/loop21 /dev/loop22
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/testraid started.
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid0
Array Size : 98304 (96.00 MiB 100.66 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Fri Dec 10 22:52:01 2021
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : -unknown-
Chunk Size : 512K
Consistency Policy : none
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 0
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
[root@nas ~]# mdadm --grow /dev/md/testraid -n 3 -l 0 --add /dev/loop23
mdadm: level of /dev/md/testraid changed to raid4
mdadm: added /dev/loop23
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid0
Array Size : 147456 (144.00 MiB 150.99 MB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Fri Dec 10 22:53:33 2021
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Chunk Size : 512K
Consistency Policy : none
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 18
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
3 7 23 2 active sync /dev/loop23
[root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0 --add /dev/loop24
mdadm: level of /dev/md/testraid changed to raid4
mdadm: added /dev/loop24
mdadm: Need to backup 6144K of critical section..
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid4
Array Size : 196608 (192.00 MiB 201.33 MB)
Used Dev Size : 49152 (48.00 MiB 50.33 MB)
Raid Devices : 5
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Fri Dec 10 22:59:56 2021
State : clean, degraded
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Chunk Size : 512K
Consistency Policy : resync
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 39
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
3 7 23 2 active sync /dev/loop23
4 7 24 3 active sync /dev/loop24
- 0 0 4 removed
[root@nas ~]# mdadm --grow /dev/md/testraid -n 5 -l 5 --add /dev/loop25
mdadm: level of /dev/md/testraid changed to raid5
mdadm: added /dev/loop25
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid5
Array Size : 196608 (192.00 MiB 201.33 MB)
Used Dev Size : 49152 (48.00 MiB 50.33 MB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Fri Dec 10 23:00:39 2021
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : parity-last
Chunk Size : 512K
Consistency Policy : resync
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 59
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
3 7 23 2 active sync /dev/loop23
4 7 24 3 active sync /dev/loop24
5 7 25 4 active sync /dev/loop25
Notice how array stay at raid4 degraded after I add 4th drive. Why is
this happening? It doesn't happen when I added 3rd drive, it did reshape
it through raid-4 but when it was finished it returned raid-0.
I also did another test where I made extra step after adding 4th disk:
[root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0
mdadm: level of /dev/md/testraid changed to raid0
[root@nas ~]# cat /proc/mdstat
Personalities : [raid1] [raid0] [linear] [raid6] [raid5] [raid4]
md118 : active raid0 loop24[4] loop23[3] loop22[1] loop21[0]
196608 blocks super 1.2 512k chunks
And when I did add 5th drive and converted array to raid-5, it correctly
defaults layout to left-symmetric.
Is this some sort of bug or working as intented? Also why is this "Need
to backup 6144K of critical section.." happening when I add 4th disk but
not when I add 3rd disk?
distro: Fedora 35
kernel: 5.15.6
mdadm: v4.2-rc2
// JiiPee
Hi Jani,
On Sat, 11 Dec 2021 00:25:14 +0200
Jani Partanen [off-list ref] wrote:
Hello,
I recently migrated from btrfs raid-5 to md raid-5 and because I
didn't have enough drives to go directly md raid-5, I basicly did
this:
- 2 disks raid-0
- added 3rd disk with grow when I had moved data, still raid-0
- added 4th disk with grow when I had moved more data, still raid-0
- added 5th disk with grow and converted to raid-5
Everything went fine other than I could not add internal bitmap
because array was created as raid-0 originally but thats not an
issue, I'll store bitmap to my /boot, it's raid-1 on SSD and ext4.
Today I noticed that my array parity layout is parity-last and I
started to wonder why is that, then I did remember that there was
some strange things happening when I did add 4th drive and I made
testing and was able to replicate what happened.
[root@nas ~]# mdadm -C /dev/md/testraid -l 0 -n 2 -N testraid
/dev/loop21 /dev/loop22
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/testraid started.
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid0
Array Size : 98304 (96.00 MiB 100.66 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Fri Dec 10 22:52:01 2021
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : -unknown-
Chunk Size : 512K
Consistency Policy : none
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 0
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
[root@nas ~]# mdadm --grow /dev/md/testraid -n 3 -l 0 --add
/dev/loop23 mdadm: level of /dev/md/testraid changed to raid4
mdadm: added /dev/loop23
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid0
Array Size : 147456 (144.00 MiB 150.99 MB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Fri Dec 10 22:53:33 2021
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Chunk Size : 512K
Consistency Policy : none
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 18
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
3 7 23 2 active sync /dev/loop23
[root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0 --add
/dev/loop24 mdadm: level of /dev/md/testraid changed to raid4
mdadm: added /dev/loop24
mdadm: Need to backup 6144K of critical section..
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid4
Array Size : 196608 (192.00 MiB 201.33 MB)
Used Dev Size : 49152 (48.00 MiB 50.33 MB)
Raid Devices : 5
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Fri Dec 10 22:59:56 2021
State : clean, degraded
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Chunk Size : 512K
Consistency Policy : resync
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 39
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
3 7 23 2 active sync /dev/loop23
4 7 24 3 active sync /dev/loop24
- 0 0 4 removed
[root@nas ~]# mdadm --grow /dev/md/testraid -n 5 -l 5 --add
/dev/loop25 mdadm: level of /dev/md/testraid changed to raid5
mdadm: added /dev/loop25
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid5
Array Size : 196608 (192.00 MiB 201.33 MB)
Used Dev Size : 49152 (48.00 MiB 50.33 MB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Fri Dec 10 23:00:39 2021
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : parity-last
Chunk Size : 512K
Consistency Policy : resync
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 59
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
3 7 23 2 active sync /dev/loop23
4 7 24 3 active sync /dev/loop24
5 7 25 4 active sync /dev/loop25
Notice how array stay at raid4 degraded after I add 4th drive. Why is
this happening? It doesn't happen when I added 3rd drive, it did
reshape it through raid-4 but when it was finished it returned raid-0.
When you are starting reshape of raid0 then array level is temporarily
changed to level4 for simple reason, reshape algorithm is already
implemented there. After migration it goes back to raid0. raid4 is
degraded because we don't have parity disk.
Reshape is managed by userspace. The process is working in background
(fork or systemd service). After the reshape it should switch
level back to raid0 but it failed in your case.
Please find out how reshape is managed in all cases (fork or
systemd service: mdadm-grow-continue). I think that there should
be only one grow process waiting/working to reshape end.
I also did another test where I made extra step after adding 4th disk:
[root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0
mdadm: level of /dev/md/testraid changed to raid0
[root@nas ~]# cat /proc/mdstat
Personalities : [raid1] [raid0] [linear] [raid6] [raid5] [raid4]
md118 : active raid0 loop24[4] loop23[3] loop22[1] loop21[0]
196608 blocks super 1.2 512k chunks
And when I did add 5th drive and converted array to raid-5, it
correctly defaults layout to left-symmetric.
If arrays stays as degraded raid4 then it is same as raid5 with
'parity-last' consistency policy, this is why you have "parity-last"
layout. It is caused by wrong level.
Is this some sort of bug or working as intented? Also why is this
"Need to backup 6144K of critical section.." happening when I add 4th
disk but not when I add 3rd disk?
It could be related with first problem, if mdadm decides to run fork,
then it could produce this message. As above, please check first how
reshape is managed in all cases.
distro: Fedora 35
kernel: 5.15.6
mdadm: v4.2-rc2
Thanks,
Mariusz