On Wed, Sep 27, 2017 at 02:56:16PM +1300, Ewen McNeill wrote:

Hi Dave,

On 27/09/17 13:43, Dave Chinner wrote:

quoted

RAID-1 does not affect the performance of the underlying volume -
the behaviour and performance of RAID-1 is identical to a single
drive, so layout can not be optimised to improve performance on
RAID-1 volumes. RAID-0, OTOH, can give dramatically better
performance if we tell the filesystem about it because we can do
things like allocate more carefully to prevent hotspots....
[...]

quoted

[why not sw=1]

Nope, because then you have no idea about how many disks you have
to spread the data over. e.g. if we have 8 disks and a sw=1, then
how do you optimise allocation to hit every disk just once for
a (su * number of disks) sized write? i.e. the sw config allows
allocation and IO sizes to be optimised to load all disks in the
RAID-0 stripe equally.

Thanks for the detailed answer.  I'd been assuming that the su/sw
values were to align with "rewritable chunks" (which clearly is
important in the RAID 5 / 6 case), and ignoring the benefit in
letting the file system choose allocation locations in RAID 0 to
best maximise distribution across the individual RAID 0 elements to
avoid hot spots.

RAID 5/6 need the same optimisations as RAID-0. In that case,
optimal RAID performance occurs when writing stripe width aligned
and sized IOs so there is no need for RMW cycles to calculate
parity. For large files being written sequentially, it really
doesn't matter that much if the head and/or tail are not exactly
stripe width aligned as things like the stripe cache in MD or the
BBWC in hardware RAID take care of delaying the data IO long enough
to do full stripe writes....

quoted

[hot spot work in late 1990s]  Out of
that came things like mkfs placing static metadata across all stripe
units instead of just the first in a stripe width, better selection
of initial alignment, etc.

It's good to hear that XFS work already anticipated the side effect
that I was concerned about (accidentally aligning everything on
"start of stripe * width boundary, and thus one disk).

I did end up creating the file system with su=512k,sw=6 (RAID 10 on
12 disks) anyway, so I'm glad to hear this is supported by earlier
performance tuning work.

As a suggestion the FAQ section (http://xfs.org/index.php/XFS_FAQ#Q:_How_to_calculate_the_correct_sunit.2Cswidth_values_for_optimal_performance)
could hint at this reasoning with, eg:

-=- cut here -=-
A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

 su = 256k
 sw = 8 (RAID-10 of 16 disks has 8 data disks)

because in RAID-10 the RAID-0 behaviour dominates performance, and this
allows XFS to spread the workload evenly across all pairs of disks.
--= cut here -=-

This assumes the reader understands exaclty how RAID works and how
the different types of RAID affect IO performance. The FAQ simply
tries to convey how to get it right without going into the
complicated reasons for doing it that way. Most users don't care
about the implementation or reasons, they just want to know how to
set it up correctly and quickly :P

and/or that FAQ entry could also talk about RAID-0 sw/su tuning
values which would provide a hint towards the "spread workload over
all disks" rationale.

The FAQ is not the place to explain how the filesystem optimises
allocation for different types of storage. The moment we add that
for RAID-0, we;ve got to do it for RAID 5, then raid 6, then
someone will ask about raid 50, etc and suddenly the typical reader
no longer understands the answer to the question....

If you look at the admin doc that is sitting a git repo:

https://git.kernel.org/pub/scm/fs/xfs/xfs-documentation.git/tree/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc

You'll see the section about all this reads:

	==== Alignment to storage geometry

	TODO: This is extremely complex and requires an entire chapter to itself.

because nobody (well, me, really) has had the time to write
everything down that is needed to cover this topic sufficiently...


Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com