AWFUL reshape speed with raid5.

I built a raid5 with 2 devices (and --assume-clean) using 2x 4GB
partitions (not logical volumes).
I then grew it to 3 devices.
The reshape speed is really really slow.

vmstat shows I/O like this:

 0  0    212  25844 141160 497484    0    0     0   612  673 1284  0  6 93  0
 0  0    212  25164 141160 497748    0    0     0    19  594 1253  1  4 95  0
 0  0    212  25044 141160 498004    0    0     0     0  374  445  0  1 99  0
 1  0    212  25220 141164 498000    0    0     0    23  506 1149  0  3 96  1
 0  0    212  25500 141164 498004    0    0     0     3  546 1416  0  5 95  0

The min/max is 1000/200000.
What might be going on here?

Kernel is 2.6.25.11 (openSUSE 11.0 x86-64 stock)

/proc/mdstat for this entry:

md99 : active raid5 sdd3[2] sdc3[1] sdb3[0]
      3903744 blocks super 1.0 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [=>...................]  reshape =  8.2% (324224/3903744)
finish=43.3min speed=1373K/sec


This is on a set of devices capable of 70+ MB/s.

No meaningful change if I start with 3 disks and grow to 4, with or
without bitmap.

-- 
Jon

What happens if you use 0.90 superblocks?

Also, are these sata ports on the mobo or sata ports on a pci-based mobo?

On Mon, 28 Jul 2008, Jon Nelson wrote:

I built a raid5 with 2 devices (and --assume-clean) using 2x 4GB
partitions (not logical volumes).
I then grew it to 3 devices.
The reshape speed is really really slow.

vmstat shows I/O like this:

0  0    212  25844 141160 497484    0    0     0   612  673 1284  0  6 93  0
0  0    212  25164 141160 497748    0    0     0    19  594 1253  1  4 95  0
0  0    212  25044 141160 498004    0    0     0     0  374  445  0  1 99  0
1  0    212  25220 141164 498000    0    0     0    23  506 1149  0  3 96  1
0  0    212  25500 141164 498004    0    0     0     3  546 1416  0  5 95  0

The min/max is 1000/200000.
What might be going on here?

Kernel is 2.6.25.11 (openSUSE 11.0 x86-64 stock)

/proc/mdstat for this entry:

md99 : active raid5 sdd3[2] sdc3[1] sdb3[0]
     3903744 blocks super 1.0 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
     [=>...................]  reshape =  8.2% (324224/3903744)
finish=43.3min speed=1373K/sec


This is on a set of devices capable of 70+ MB/s.

No meaningful change if I start with 3 disks and grow to 4, with or
without bitmap.

-- 
Jon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Mon, Jul 28, 2008 at 1:14 PM, Justin Piszcz [off-list ref] wrote:

What happens if you use 0.90 superblocks?

An even more spectacular explosion. Really quite impressive, actually.
Just a small sample of the >800 lines:

Jul 28 13:18:54 turnip kernel: itmap 9: invalid bitmap page 9: invalid
bitmap page 9: invalid bitmap p9: invalid bitmap pag9: invalid bitmap
9: invalid bitmap page9: invalid bitmap pa9: invalid bitmap pag9:
invalid bitmap 9: invalid bitmap 9: invalid bitmap pag9: invalid
bitmap page9: invalid bitmap 9: invalid bitmap 9: invalid bitmap
page9: invalid bitmap pa9: invalid bitmap pag9: invalid bitmap pa9:
invalid bitmap p9: invalid bitmap page 9: invalid bitmap page 9:
invalid bitmap p9: invalid bitmap 9: invalid bitmap page9: invalid
bitmap pag9: invalid bitmap pag9: invalid bitmap page9: invalid bitmap
9: invalid bitmap pag9: invalid bitmap page9: invalid bitmap pag9:
invalid bitmap9: invalid bitmap page9: invalid bitmap pag9: invalid
bitmap pa9: invalid bitmap page9: invalid bitmap pa9: invalid bitmap
p9: invalid bitmap page 9: invalid bitmap page 9: invalid bit9:
invalid bitmap pa9: invalid bitmap page 9: invalid bitmap pag9:
invalid bitmap page9: invalid bitmap 9: invalid bitmap pa9: invalid
bitmap page9: invalid

(Note the page -> pag -> p ... variations/corruption/whatever).

and

Jul 28 13:18:55 turnip kernel: i9: i9: 9: 9: i9: 9: i9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: i9: 9: 9: 9: 9: 9: 9: 9: 9: i9: i9: 9: i9: 9: i9:
i9: 9: 9: i9: 9: i9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: i9: 9: 9: 9: 9:
9: 9: 9: 9: i9: i9: 9: in9: 9: 9: 9: 9: 9: i9: 9: i9: 9: i9: 9: i9:
i9: 9: 9: 9: 9: 9: 9: i9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: i9: 9: i9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: i9: i9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: i9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: i9: i9: 9: 9: 9: 9: 9: 9: i9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
Jul 28 13:18:55 turnip kernel:  9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: i9: i9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: i9: i9: 9: 9:
9: 9: 9: 9: 9: i9: 9: i9: 9: i9: 9: i9: 9: i9: 9: i9: 9: i9: 9: i9: 9:
i9: 9: i9: 9: i9: 9: i9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: i9: 9: i9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9:
9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: 9: invalid bitmap page request:
222 (> 122)

Also, are these sata ports on the mobo or sata ports on a pci-based mobo?

The ports are on a PCI-E motherboard, MCP55 chipset. The
motherboard/chipset/whatever is *not* the problem.

-- 
Jon

Some more data points, observations, and questions.

For each test, I'd --create the array, drop the caches, --grow, and
then watch vmstat and also record the time between

kernel: md: resuming resync of md99 from checkpoint.
and
kernel: md: md99: resync done.

I found two things:

1. metadata version matters. Why?
2. VERY LITTLE I/O takes place (between 0 and 100KB/s, typically no
I/O at all) according to vmstat. Why? If it takes 1m34s to "grow" the
array, but no I/O is taking place, then what is actually taking so
long?
3. I removed the bitmap for these tests. Having a bitmap meant that
the overall speed was REALLY HORRIBLE.

The results:

metadata: time taken

0.9: 27s
1.0: 27s
1.1: 37s
1.2: 1m34s

Questions (repeated):

1. Why does the metadata version matter so much?
2. If no I/O is taking place, why does it take so long? [ NOTE: I/O
must be taking place but why doesn't vmstat show it? ]

-- 
Jon

Jon Nelson wrote:

Some more data points, observations, and questions.

For each test, I'd --create the array, drop the caches, --grow, and
then watch vmstat and also record the time between

kernel: md: resuming resync of md99 from checkpoint.
and
kernel: md: md99: resync done.

I found two things:

1. metadata version matters. Why?
2. VERY LITTLE I/O takes place (between 0 and 100KB/s, typically no
I/O at all) according to vmstat. Why? If it takes 1m34s to "grow" the
array, but no I/O is taking place, then what is actually taking so
long?

I *think* that internal md io is not being shown.

I know I can tell an array to check itself, and have mdstat indicate a speed of 
35MB/second and vmstat indicates no IO  was happening.   The same happens when 
an array is rebuilding, vmstat indicates no IO.   If you do IO to the md device 
from outside it does show that.   And in both cases visually checking the 
confirms that quite a lot appears to be going on.

                               Roger

There once was a bug in an earlier kernel, in which the min_speed is what 
the rebuild ran at if you had a specific chunk size, have you tried to 
echo 30000 > to min_speed?  Does it increase it to 30mb/s for the rebuild?

On Mon, 28 Jul 2008, Jon Nelson wrote:

Some more data points, observations, and questions.

For each test, I'd --create the array, drop the caches, --grow, and
then watch vmstat and also record the time between

kernel: md: resuming resync of md99 from checkpoint.
and
kernel: md: md99: resync done.

I found two things:

1. metadata version matters. Why?
2. VERY LITTLE I/O takes place (between 0 and 100KB/s, typically no
I/O at all) according to vmstat. Why? If it takes 1m34s to "grow" the
array, but no I/O is taking place, then what is actually taking so
long?
3. I removed the bitmap for these tests. Having a bitmap meant that
the overall speed was REALLY HORRIBLE.

The results:

metadata: time taken

0.9: 27s
1.0: 27s
1.1: 37s
1.2: 1m34s

Questions (repeated):

1. Why does the metadata version matter so much?
2. If no I/O is taking place, why does it take so long? [ NOTE: I/O
must be taking place but why doesn't vmstat show it? ]

-- 
Jon

-----Original Message-----
From: linux-raid-owner@vger.kernel.org

[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Justin Piszcz

Sent: Monday, July 28, 2008 2:44 PM
To: Jon Nelson
Cc: LinuxRaid
Subject: Re: AWFUL reshape speed with raid5.

There once was a bug in an earlier kernel, in which the min_speed is

what 

the rebuild ran at if you had a specific chunk size, have you tried to 
echo 30000 > to min_speed?  Does it increase it to 30mb/s for the

rebuild?

On Mon, 28 Jul 2008, Jon Nelson wrote:

quoted

Some more data points, observations, and questions.

For each test, I'd --create the array, drop the caches, --grow, and
then watch vmstat and also record the time between

kernel: md: resuming resync of md99 from checkpoint.
and
kernel: md: md99: resync done.

I found two things:

1. metadata version matters. Why?
2. VERY LITTLE I/O takes place (between 0 and 100KB/s, typically no
I/O at all) according to vmstat. Why? If it takes 1m34s to "grow" the
array, but no I/O is taking place, then what is actually taking so
long?
3. I removed the bitmap for these tests. Having a bitmap meant that
the overall speed was REALLY HORRIBLE.

The results:

metadata: time taken

0.9: 27s
1.0: 27s
1.1: 37s
1.2: 1m34s

Questions (repeated):

1. Why does the metadata version matter so much?
2. If no I/O is taking place, why does it take so long? [ NOTE: I/O
must be taking place but why doesn't vmstat show it? ]

-- 
Jon

You are incorrectly working from the premise that vmstat is measures
disk activity.  It does not.  Vmstat has no idea how many actual bytes
get sent to, or received from disk drives.

Why not do a real test and hook up a pair of SAS, SCSI, or FC disks,
then issue some LOG SENSE commands to report the actual number of bytes
read & written to each disk during the rebuild? If the disks are
FibreChannel, then you have even more ways to measure true throughput in
bytes.  It will not be an estimate, it will be a real count of
cumulative bytes read, written, re-read/re-written, recovered, etc., for
any instant in time.  Heck, if you have Seagate and some other disks,
then you can even see detailed information for cached reads so you can
see if any particular md configuration results in a higher number of
cached I/Os, meaning greater efficiency and smaller overall latency.


David @ santools dot com

David

On Mon, Jul 28, 2008 at 2:59 PM, David Lethe [off-list ref] wrote:

quoted

-----Original Message-----
From: linux-raid-owner@vger.kernel.org

[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Justin Piszcz

quoted

Sent: Monday, July 28, 2008 2:44 PM
To: Jon Nelson
Cc: LinuxRaid
Subject: Re: AWFUL reshape speed with raid5.

There once was a bug in an earlier kernel, in which the min_speed is

what

quoted

the rebuild ran at if you had a specific chunk size, have you tried to
echo 30000 > to min_speed?  Does it increase it to 30mb/s for the

rebuild?

As I said in my original post, I'm running 2.6.25.11

You are incorrectly working from the premise that vmstat is measures
disk activity.  It does not.  Vmstat has no idea how many actual bytes
get sent to, or received from disk drives.

Fair enough.

Why not do a real test and hook up a pair of SAS, SCSI, or FC disks,

...

Cost.

-- 
Jon

Jon Nelson wrote:

1. metadata version matters. Why?
2. VERY LITTLE I/O takes place (between 0 and 100KB/s, typically no
I/O at all) according to vmstat. Why? If it takes 1m34s to "grow" the
array, but no I/O is taking place, then what is actually taking so
long?
3. I removed the bitmap for these tests. Having a bitmap meant that
the overall speed was REALLY HORRIBLE.

The results:

metadata: time taken

0.9: 27s
1.0: 27s
1.1: 37s
1.2: 1m34s

Questions (repeated):

1. Why does the metadata version matter so much?
  

I have no idea.

2. If no I/O is taking place, why does it take so long? [ NOTE: I/O
must be taking place but why doesn't vmstat show it? ]
  

vmstat doesn't tell you enough, you need a tool to show per-device and 
per-partition io, which will give you what you need. I can't put a 
finger on the one I wrote, but there are others.


-- 
Bill Davidsen [off-list ref]
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

On Wed, Jul 30, 2008 at 11:23 AM, Bill Davidsen [off-list ref] wrote:

Jon Nelson wrote:

quoted

2. If no I/O is taking place, why does it take so long? [ NOTE: I/O
must be taking place but why doesn't vmstat show it? ]

vmstat doesn't tell you enough, you need a tool to show per-device and
per-partition io, which will give you what you need. I can't put a finger on
the one I wrote, but there are others.

I gave dstat a try (actually, I rather prefer dstat over vmstat...):

This is right before, during, and after the --grow operation.

--dsk/sdb-----dsk/sdc-----dsk/sdd-----dsk/sde----dsk/total-
 read  writ: read  writ: read  writ: read  writ: read  writ
   0    44k:   0    24k:   0    24k:   0     0 :   0   184k
   0     0 :   0     0 :   0     0 :   0     0 :   0     0
   0    24k:   0     0 :   0     0 :   0     0 :   0    48k
  32M   14k:  32M 2048B:  32M 2048B:   0     0 : 191M   36k
  63M    0 :  63M    0 :  63M    0 :   0     0 : 377M    0
  65M    0 :  65M    0 :  65M    0 :   0     0 : 391M    0
  72M    0 :  73M    0 :  73M    0 :   0     0 : 437M    0
  74M    0 :  73M    0 :  74M    0 :   0     0 : 441M    0
  70M   48k:  70M    0 :  70M    0 :   0     0 : 419M   96k
  61M   44k:  61M   16k:  62M   32k:   0     0 : 368M  184k
  71M    0 :  72M    0 :  71M    0 :   0     0 : 429M    0
  74M    0 :  73M    0 :  73M    0 :   0     0 : 439M    0
  73M    0 :  73M    0 :  73M    0 :   0     0 : 439M    0
  71M   20k:  71M    0 :  71M    0 :   0     0 : 426M   40k
  72M    0 :  72M    0 :  73M    0 :   0     0 : 434M    0
  73M    0 :  74M    0 :  73M    0 :   0     0 : 442M    0
  60M   40k:  59M   16k:  59M   28k:   0     0 : 356M  168k
  73M    0 :  73M    0 :  73M    0 :   0     0 : 438M    0
  70M   24k:  69M    0 :  70M    0 :   0     0 : 418M   48k
  72M    0 :  71M    0 :  72M    0 :   0     0 : 430M    0
  73M    0 :  73M    0 :  73M    0 :   0     0 : 437M    0
  71M    0 :  71M    0 :  71M    0 :   0     0 : 427M    0
  73M    0 :  73M    0 :  73M    0 :   0     0 : 437M    0
  71M   24k:  71M    0 :  71M    0 :   0     0 : 428M   48k
  72M    0 :  72M    0 :  72M    0 :   0     0 : 432M    0
  73M    0 :  73M    0 :  73M    0 :   0     0 : 437M    0
  71M 4096B:  70M    0 :  70M    0 :   0     0 : 422M 8192B
  58M    0 :  58M    0 :  58M    0 :   0     0 : 350M    0
  59M   24k:  60M    0 :  59M    0 :   0     0 : 357M   48k
  74M    0 :  73M    0 :  74M    0 :   0     0 : 441M    0
  19M 8192B:  19M 8192B:  19M 4096B:   0     0 : 114M   40k
   0     0 :   0     0 :   0     0 :   0     0 :   0     0
   0     0 :   0     0 :   0     0 :   0     0 :   0     0
   0   160k:   0    16k:   0    20k:   0     0 :   0   392k

So. Clearly, lots of I/O. 440MB/s total.  Almost entirely reads.


Question:  to --grow --size the array, clearly we see lots of reads.
Why aren't we seeing any (meaningful) writes? If there are no writes,
then what purpose do the reads serve?

Bill Davidsen wrote:

vmstat doesn't tell you enough, you need a tool to show per-device and
per-partition io, which will give you what you need. I can't put a
finger on the one I wrote, but there are others.

iostat?

David

On Wed, 30 Jul 2008, Jon Nelson wrote:

On Wed, Jul 30, 2008 at 11:23 AM, Bill Davidsen [off-list ref] wrote:

quoted

Jon Nelson wrote:

quoted

2. If no I/O is taking place, why does it take so long? [ NOTE: I/O
must be taking place but why doesn't vmstat show it? ]

vmstat doesn't tell you enough, you need a tool to show per-device and
per-partition io, which will give you what you need. I can't put a finger on
the one I wrote, but there are others.

I gave dstat a try (actually, I rather prefer dstat over vmstat...):

This is right before, during, and after the --grow operation.

--dsk/sdb-----dsk/sdc-----dsk/sdd-----dsk/sde----dsk/total-
read  writ: read  writ: read  writ: read  writ: read  writ
  0    44k:   0    24k:   0    24k:   0     0 :   0   184k
  0     0 :   0     0 :   0     0 :   0     0 :   0     0
  0    24k:   0     0 :   0     0 :   0     0 :   0    48k
 32M   14k:  32M 2048B:  32M 2048B:   0     0 : 191M   36k
 63M    0 :  63M    0 :  63M    0 :   0     0 : 377M    0
 65M    0 :  65M    0 :  65M    0 :   0     0 : 391M    0
 72M    0 :  73M    0 :  73M    0 :   0     0 : 437M    0
 74M    0 :  73M    0 :  74M    0 :   0     0 : 441M    0
 70M   48k:  70M    0 :  70M    0 :   0     0 : 419M   96k
 61M   44k:  61M   16k:  62M   32k:   0     0 : 368M  184k
 71M    0 :  72M    0 :  71M    0 :   0     0 : 429M    0
 74M    0 :  73M    0 :  73M    0 :   0     0 : 439M    0
 73M    0 :  73M    0 :  73M    0 :   0     0 : 439M    0
 71M   20k:  71M    0 :  71M    0 :   0     0 : 426M   40k
 72M    0 :  72M    0 :  73M    0 :   0     0 : 434M    0
 73M    0 :  74M    0 :  73M    0 :   0     0 : 442M    0
 60M   40k:  59M   16k:  59M   28k:   0     0 : 356M  168k
 73M    0 :  73M    0 :  73M    0 :   0     0 : 438M    0
 70M   24k:  69M    0 :  70M    0 :   0     0 : 418M   48k
 72M    0 :  71M    0 :  72M    0 :   0     0 : 430M    0
 73M    0 :  73M    0 :  73M    0 :   0     0 : 437M    0
 71M    0 :  71M    0 :  71M    0 :   0     0 : 427M    0
 73M    0 :  73M    0 :  73M    0 :   0     0 : 437M    0
 71M   24k:  71M    0 :  71M    0 :   0     0 : 428M   48k
 72M    0 :  72M    0 :  72M    0 :   0     0 : 432M    0
 73M    0 :  73M    0 :  73M    0 :   0     0 : 437M    0
 71M 4096B:  70M    0 :  70M    0 :   0     0 : 422M 8192B
 58M    0 :  58M    0 :  58M    0 :   0     0 : 350M    0
 59M   24k:  60M    0 :  59M    0 :   0     0 : 357M   48k
 74M    0 :  73M    0 :  74M    0 :   0     0 : 441M    0
 19M 8192B:  19M 8192B:  19M 4096B:   0     0 : 114M   40k
  0     0 :   0     0 :   0     0 :   0     0 :   0     0
  0     0 :   0     0 :   0     0 :   0     0 :   0     0
  0   160k:   0    16k:   0    20k:   0     0 :   0   392k

So. Clearly, lots of I/O. 440MB/s total.  Almost entirely reads.


Question:  to --grow --size the array, clearly we see lots of reads.
Why aren't we seeing any (meaningful) writes? If there are no writes,
then what purpose do the reads serve?

In dstat, the speed is doubled for the total for some reason, divide by 
2 (if you compare with iostat -x -k 1) you should see the difference.

As far as the grow itself, the last time I did it was 2-3 years ago but if 
I recall it ran for 24hrs+ (IDE system, PCI, etc) between 5-15MiB/s.

Justin.

David Greaves wrote:

Bill Davidsen wrote:
  

quoted

vmstat doesn't tell you enough, you need a tool to show per-device and
per-partition io, which will give you what you need. I can't put a
finger on the one I wrote, but there are others.
    

iostat?
  

dstat seems to do what he wants, the one I wrote produced a file which 
could be used by gnuplot to generate neat graphical output to make 
problems glaringly obvious.

-- 
Bill Davidsen [off-list ref]
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

On Wed, Jul 30, 2008 at 12:08 PM, Justin Piszcz [off-list ref] wrote:

In dstat, the speed is doubled for the total for some reason, divide by 2
(if you compare with iostat -x -k 1) you should see the difference.

Ah, nice catch. It still doesn't change my question.
Upon further reflection, I believe I know what has caused the
read/write disparity:

1. I've done so much testing with these devices that the contents have
been zeroed many times.
2. I am GUESSING that if the raid recovery code reads from drives A,
B, and C, builds the appropriate checksum and verifies it, if the
checksum matches it skips the write. I believe that this is what is
happening. To confirm, I wrote a few gigs of /dev/urandom to one of
the devices and re-tested. Indeed, this time around I saw plenty of
writing. One mystery solved.

Remaining questions:

1. Why does the version of metadata matter so much in a --grow --size operation?
2. There appear to be bugs when a bitmap is used. Can somebody else confirm?
3. I'll look into the awful speed thing later as that doesn't seem to
be an issue.

-- 
Jon

On Mon, Jul 28, 2008 at 12:39 PM, Jon Nelson
[off-list ref] wrote:

I built a raid5 with 2 devices (and --assume-clean) using 2x 4GB
partitions (not logical volumes).
I then grew it to 3 devices.
The reshape speed is really really slow.

vmstat shows I/O like this:

 0  0    212  25844 141160 497484    0    0     0   612  673 1284  0  6 93  0
 0  0    212  25164 141160 497748    0    0     0    19  594 1253  1  4 95  0
 0  0    212  25044 141160 498004    0    0     0     0  374  445  0  1 99  0
 1  0    212  25220 141164 498000    0    0     0    23  506 1149  0  3 96  1
 0  0    212  25500 141164 498004    0    0     0     3  546 1416  0  5 95  0

The min/max is 1000/200000.
What might be going on here?

Kernel is 2.6.25.11 (openSUSE 11.0 x86-64 stock)

/proc/mdstat for this entry:

md99 : active raid5 sdd3[2] sdc3[1] sdb3[0]
     3903744 blocks super 1.0 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
     [=>...................]  reshape =  8.2% (324224/3903744)
finish=43.3min speed=1373K/sec


This is on a set of devices capable of 70+ MB/s.

I found some time to give this another shot.
It's still true!

Here is how I built the array:

mdadm --create /dev/md99 --level=raid5 --raid-devices=2
--spare-devices=0 --assume-clean --metadata=1.0 --chunk=64 /dev/sdb3
/dev/sdc3

and then I added a drive:

mdadm --add /dev/md99 /dev/sdd3

and then I grew the array to 3 devices:

mdadm --grow /dev/md99 --raid-devices=3

This is what the relevant portion of /proc/mdstat looks like:

md99 : active raid5 sdd3[2] sdc3[1] sdb3[0]
      3903744 blocks super 1.0 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [=>...................]  reshape =  6.1% (241920/3903744)
finish=43.0min speed=1415K/sec

The 1000/200000 min/max defaults are being used.
If I bump up the min to, say, 30000, the rebuild speed does grow to
hover around 30000.

As Justin Piszcz said:

There once was a bug in an earlier kernel, in which the min_speed is
what the rebuild ran at if you had a specific chunk size, have you
tried to echo 30000 > to min_speed?  Does it increase it to 30mb/s for
the rebuild?

Yes, apparently, it does. However, 'git log drivers/md' in the
linux-2.6 tree doesn't show anything obvious for me. Can somebody
point me to a specific commit, patch, etc... because as of 2.6.25.11
it's apparently still a problem (on an otherwise idle system, too).

No meaningful change if I start with 3 disks and grow to 4, with or
without bitmap.

--
Jon

-- 
Jon