Re: RAID10 Write Performance

On Thu, Dec 24 2015, Marc Smith wrote:

Okay, thanks, I'll turn it back on and try some different chunk sizes.

For my own knowledge, why/what is taking place under the covers that
causes this behavior? When testing with fio, it sometimes takes 1-2
minutes of "ramp up time" before the performance numbers are
good/expected (when the write-intent bitmap is enabled).

Whenever md needs to write to a region (a bitmap-chunk) of the array
that it hasn't written to recently, it needs to set a bit and write out
the bitmap first.  It tries to gather multiple writes together and set
several bits at once, but a synchronous workload will defeat that.
Once the bit is set it will stay set until several seconds after the
last write.  I think it defaults to 5 seconds.
  mdadm -X /dev/some-component
will list it as 'daemon sleep'.

So the delay you are seeing is the time it takes to get all of those
bits set.  1 minute does sound like a long time, though it the writes
are synchronous that would easily explain it.
With a larger chunk size, there are fewer bits to set so fewer time that
the drives need to seek to the other end of the disk to write out the
bitmap.

If you run "watch -n 0.1 mdadm -X /dev/something" in a window it will
report how many bits are set moment by moment.  That might give you some
feel for what is happening.

NeilBrown

Thanks,

Marc


On Tue, Dec 22, 2015 at 9:20 PM, NeilBrown [off-list ref] wrote:

quoted

On Wed, Dec 23 2015, Marc Smith wrote:

quoted

Solved... appears it was the write-intent bitmap that caused the
performance issues. I discovered if I left the test running longer
than 60 seconds, the performance would eventually climb to where I'd
expect it. I ran 'mdadm --grow --bitmap=none /dev/md0' and now random
write performance is high/good/stable right off the bat.

Keeping a write-intent bitmap really is a good idea.
Using a larger bitmap chunk size can reduce the performance penalty and
preserve much of the value.  It is easy enough to experiment with
different sizes.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html