On 11/11/2009 06:32 PM, Ben DJ wrote:

Hi,

I didn't want to hijack the thread, so a new one here.

On Wed, Nov 11, 2009 at 2:10 PM, Robin Hill [off-list ref] wrote:

quoted

It's true for RAID-10, yes.  You can't physically grow the array, but
you can definitely add/remove the bitmap.

Thanks for clearing that up.  The manpage is a bit unclear to my read.

I've just been reading threads about proper sizing.  Large
bitmap-chunk seems good, larger than "1 million bits" -- not, and an
old bug (resolved?) if bitmap-chunk is smaller than raid10 chunk size.

I've two arrays --

   /dev/md0 (RAID-1, across 4x 160MB partitions)

   /dev/md1 (RAID-10/f2 , across 4x ~1TB partitions, --chunk=256 )

The first array is so small, that resync takes just a few seconds
anyway.  Is there any advantage to still installing an internal
write-intent bitmap on it?

Not in my opinion.  I skip bitmaps on boot arrays and other smallish
arrays like that.

The second array takes a few hours to resync from scratch, and so the
bitmap has performance value.  What's the right size for
--bitmap-chunk for an internal bitmap?  Iiuc, the default that "is
automatically deteremined to make best use of available space" results
in 2x-4x (some say 10%) write-performance slowdowns.

It makes for noticeable slowdowns anyway.  How bad is dependent on your
data writing patterns.  Lots of random writes will actually show a
larger slowdown than more sequential writing.  The main thing to
understand is that a bitmap like this is useless if the raid stack
doesn't stop any write going to the disk unless the bit for that write's
sector is set to dirty.  So, when a new write is initiated on a clean
array, the write is help up until the bitmap write to dirty the proper
bits completes, and only then can the normal write proceed.  So, with
lots of random I/O, or even with sequential I/O on very small bitmap
chunk sizes, you end up spending a significant amount of time holding up
writes as you dirty the bits on disk.  Picking a larger bitmap chunk
helps to increase the likelihood that more writes will stream without
having to wait on a new bitmap dirty write.

Given that the only real benefit to the bitmap is reduced resync time in
the event something happens, and given that as you said a 160MB section
of array can resync in a relatively short time, and given that a smaller
bitmap chunk hurts performance *all* the time versus only helping in
rare circumstances, bigger is better in my opinion.

I haven't done specific testing of performance differences with
different size bitmap chunks, but my seat of the pants review puts the
32768 area as a good starting point.  Any given chunk will resync in
just a second or so, but it doesn't cause as much performance slowdown
as the default chunk size.


-- 
Doug Ledford [off-list ref]
              GPG KeyID: CFBFF194
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband