Re: unbelievably bad performance: 2.6.27.37 and raid6

On Tue, November 3, 2009 5:09 pm, Michael Evans wrote:

On Mon, Nov 2, 2009 at 9:36 PM, NeilBrown [off-list ref] wrote:

quoted

On Tue, November 3, 2009 2:03 am, Jon Nelson wrote:

quoted

On Mon, Nov 2, 2009 at 8:54 AM, Bill Davidsen [off-list ref] wrote:

quoted

Jon Nelson wrote:

quoted

I have a 4 disk raid6. The disks are individually capable of (at
least) 75MB/s on average.
The raid6 looks like this:

md0 : active raid6 sda4[0] sdc4[5] sdd4[4] sdb4[6]
     613409536 blocks super 1.1 level 6, 64k chunk, algorithm 2 [4/4]
[UUUU]

The raid serves basically as an lvm physical volume.

While rsyncing a file from an ext3 filesystem to a jfs filesystem, I
am observing speeds in the 10-15MB/s range.
That seems really really slow.

It is really slow, recent kernels seem to be unsuitable for use as
large
file servers, as the performance is, as you described it,
"unbelievably
bad."

Yeah. I'm hoping that the 2.6.31.XX stable kernel series gets some of
these improvements, the .27 series has been not the most stable for me
either.  2.6.27.25 was the last rock-solid of the .27 series for me.

I wouldn't get your hopes up...
I did some limited testing of simple writes to ext2 and the current
32-pre kernel is noticably slower than .26 .27 .28 .29 .. (that is as
far as I got with testing... I should write a script and leave it
running
overnight to get a broader picture).

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Maybe my speed results even after the 'fix' are also this issue:  I
expect each of my drives is capable of at least 8MB/sec sustained
(highly pessimistic).

     2909829120 blocks super 1.1 level 6, 128k chunk, algorithm 18
[8/8] [UUUUUUUU]
     [==>..................]  reshape = 10.4% (50708096/484971520)
finish=3989.0min speed=1813K/sec

The 'backup file' is on a separate raid 1 device and approximately 25
mb in size.  My cpu has virtually no load and I've got gigs of memory
free.

A reshape is a fundamentally slow operation.  Each block needs to
be read and then written somewhere else so there is little opportunity
for streaming.
An in-place reshape (i.e the array doesn't get bigger or smaller) is
even slower as we have to take a backup copy of each range of blocks
before writing them back out.  This limits streaming even more.

It is possible to get it fast than it is by increasing the
array's stripe_cache_size and also increasing the 'backup' size
that mdadm uses.  mdadm-3.1.1 will try to do better in this respect.
However it will still be significantly slower than e.g. a resync.

So reshape will always be slow.  It is a completely different issue
to filesystem activity on a RAID array being slow.  Recent reports of
slowness are, I think, not directly related to md/raid.  It is either
the filesystem or the VM or a combination of the two that causes
these slowdowns.


NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html