md raid6: slow resync if fs is mounted rw

Dear list,

I am seeing a strange performance issue on a 4 x 4TB md raid6 on kernel
3.12. Initial resync goes nice and fast at 130MB/s:

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      7751269376 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
      [=>...................]  resync =  7.5% (293925504/3875634688) finish=458.4min speed=130210K/sec

md0 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      29280128 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>

...unless the ext4 filesystem on the array is mounted read-write!
Then, speed plummets to 1 - 7MB/s. The filesystem is empty and totally
idle; a simple "mount -o remount,ro" restores the speed to 130MB/s.

I dumped the /sys/block/md1/md tree with the filesystem mounted ro and
rw, the diff looks like this:

So, stripe_cache_active is zero if the fs is mounted rw. Hmm.

Interestingly, if I increase sync_speed_min to 100MB/s, I do get 100MB/s.

Any idea what is going on?

Best regards,
Jakob Unterwurzacher

On Tue, 25 Feb 2014 23:55:15 +0100 Jakob Unterwurzacher [off-list ref]
wrote:

Dear list,

I am seeing a strange performance issue on a 4 x 4TB md raid6 on kernel
3.12. Initial resync goes nice and fast at 130MB/s:

quoted

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      7751269376 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
      [=>...................]  resync =  7.5% (293925504/3875634688) finish=458.4min speed=130210K/sec

md0 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      29280128 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>

...unless the ext4 filesystem on the array is mounted read-write!
Then, speed plummets to 1 - 7MB/s. The filesystem is empty and totally
idle; a simple "mount -o remount,ro" restores the speed to 130MB/s.

I dumped the /sys/block/md1/md tree with the filesystem mounted ro and
rw, the diff looks like this:

quoted

# diff -u md.rw md.ro
[...]

--- md.rw/stripe_cache_active   2014-02-25 17:33:31.250019324 +0100
+++ md.ro/stripe_cache_active   2014-02-25 17:33:31.250019324 +0100

@@ -1 +1 @@
-0
+256

diff -u md.rw/sync_speed md.ro/sync_speed

--- md.rw/sync_speed    2014-02-25 17:33:31.250019324 +0100
+++ md.ro/sync_speed    2014-02-25 17:33:31.250019324 +0100

@@ -1 +1 @@
-1052
+126244

So, stripe_cache_active is zero if the fs is mounted rw. Hmm.

Interestingly, if I increase sync_speed_min to 100MB/s, I do get 100MB/s.

Any idea what is going on?

Simplest explanation would be that ext4 is writing to the array
occasionally.  Maybe 1 block per second.  That would be enough for md to
decide that the array is busy and to back-off the resync.

Try
  blktrace -w 20 /dev/md1

while the filesystem is mounted read-write, then
  blkparse md1.blktrace*

do we if anything writes are going to the array.

NeilBrown

On 26/02/14 06:55, Jakob Unterwurzacher wrote:

Dear list,

I am seeing a strange performance issue on a 4 x 4TB md raid6 on kernel
3.12. Initial resync goes nice and fast at 130MB/s:

quoted

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
       7751269376 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
       [=>...................]  resync =  7.5% (293925504/3875634688) finish=458.4min speed=130210K/sec

md0 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
       29280128 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>

...unless the ext4 filesystem on the array is mounted read-write!
Then, speed plummets to 1 - 7MB/s. The filesystem is empty and totally
idle; a simple "mount -o remount,ro" restores the speed to 130MB/s.

You are probably expeririencing an effect of this :

http://www.thomas-krenn.com/en/wiki/Ext4_Filesystem#Lazy_Initialization

Regards,
Brad

On Wed, Feb 26, 2014 at 3:31 AM, Brad Campbell
[off-list ref] wrote:

You are probably expeririencing an effect of this :

http://www.thomas-krenn.com/en/wiki/Ext4_Filesystem#Lazy_Initialization

That's it. ext4lazyinit shows up in the blktrace and also flashes up
every now and then in iotop - I did not notice it before.
I reformatted with "mkfs.ext4 -E
lazy_itable_init=0,lazy_journal_init=0" and the issue is gone. Of
course, just waiting a bit with the filesystem mounted rw would have
done it, too.

Thanks a lot Neil and Brad!
Jakob