From: Dave Chinner <david@fromorbit.com> Date: 2008-08-25 01:59:35
On Fri, Aug 22, 2008 at 12:29:10PM +1000, Nick Piggin wrote:
On Friday 22 August 2008 03:08, Dave Chinner wrote:
quoted
On Thu, Aug 21, 2008 at 07:33:34PM +1000, Nick Piggin wrote:
quoted
quoted
I don't really see it as too complex. If you know how you want the
request to be handled, then it should be possible to implement.
That is the problem in a nutshell. Nobody can keep up with all
the shiny new stuff that is being implemented,let alone the
subtle behavioural differences that accumulate through such
change...
I'm not sure exactly what you mean.. I certainly have not been keeping
up with all the changes here as I'm spending most of my time on other
things lately...
But from what I see, you've got a fairly good handle on analysing the
elevator behaviour (if only the end result).
Only from having to do this analysis over and over again trying to
understand what has changed in the elevator that has negated the
effect of some previous optimisation....
So if you were to tell
Jens that "these blocks" need more priority, or not to contribute to
a process's usage quota, etc. then I'm sure improvements could be
made.
It's exactly this sort of complexity that is the problem. When the
behaviour of such things change, filesystems that are optimised for
the previous behaviour are not updated - we're not even aware that
the elevator has been changed in some subtle manner that breaks
the optimisations that have been done.
To keep on top of this, we keep adding new variations and types and
expect the filesystems to make best use of them (without
documentation) to optimise for certain situations. Example - the
new(ish) BIO_META tag that only CFQ understands. I can change the
way XFS issues bios to use this tag to make CFQ behave the same way
it used to w.r.t. metadata I/O from XFS, but then the deadline and
AS will probably regress because they don't understand that tag and
still need the old optimisations that just got removed. Ditto for
prioritised bio dispatch - CFQ supports it but none of the others
do.
IOWs, I am left with a choice - optimise for a specific elevator
(CFQ) to the detriment of all others (noop, as, deadline), or make
the filesystem work best with the simple elevator (noop) and
consider the smarter schedulers deficient if they are slower than
the noop elevator....
Or am I completely misunderstanding you? :)
You're suggesting that I add complexity to solve the too much complexity
problem.... ;)
quoted
quoted
quoted
With the way the elevators have been regressing,
improving and changing behaviour,
AFAIK deadline, AS, and noop haven't significantly changed for years.
Yet they've regularly shown performance regressions because other
stuff has been changing around them, right?
Is this rhetorical? Because I don't see how *they* could be showing
regular performance regressions.
I get private email fairly often asking questions as to why XFS is
slower going from, say, 2.6.23 to 2.6.24 and then speeds back up in
2.6.25. I seen a number of cases where the answer to this was that
elevator 'x' with XFS in 2.6.x because for some reason it is much,
much slower than the others on that workload on that hardware.
As seen earlier in this thread, this can be caused by a problem with
the hardware, firmware, configuration, driver bugs, etc - there are
so many combinations of variables that can cause performance issues
that often the only 'macro' level change that you can make to avoid
them is to switch schedulers. IOWs, while a specific scheduler has
not changed, the code around it has changed sufficiently for a
specific elevator to show a regression compared to the otherr
elevators.....
Basically, the complexity of the interactions between the
filesystems, elevators and the storage devices is such that there
are transient second order effects occurring that are not reported
widely because they are easily worked around by switching elevators.
Deadline literally had its last
behaviour change nearly a year ago, and before that was before
recorded (git) history.
AS hasn't changed much more frequently, although I will grant that it
and CFS add a lot more complexity. So I would always compare results
with deadline or noop.
Which can still change by things like changing merging behaviour.
Granted, it is less complex, but still we can have subtle changes
having major impact in less commonly run workloads...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
To keep on top of this, we keep adding new variations and types and
expect the filesystems to make best use of them (without
documentation) to optimise for certain situations. Example - the
new(ish) BIO_META tag that only CFQ understands. I can change the
way XFS issues bios to use this tag to make CFQ behave the same way
it used to w.r.t. metadata I/O from XFS, but then the deadline and
AS will probably regress because they don't understand that tag and
still need the old optimisations that just got removed. Ditto for
prioritised bio dispatch - CFQ supports it but none of the others
do.
There's nothing wrong with adding BIO_META (for example) and other
hints in _principle_. You should be able to ignore it with no adverse
effects. If its not used by a filesystem (and there's nothing else
competing to use the same disk), I would hope to see the same
performance as other kernels which don't have it.
If the elevators are being changed in such a way that old filesystem
code which doesn't use new hint bits is running significantly slower,
surely that's blatant elevator regression, and that's where the bugs
should be reported and fixed?
-- Jamie
From: Dave Chinner <david@fromorbit.com> Date: 2008-08-26 03:08:09
On Mon, Aug 25, 2008 at 01:01:47PM +0100, Jamie Lokier wrote:
Dave Chinner wrote:
quoted
To keep on top of this, we keep adding new variations and types and
expect the filesystems to make best use of them (without
documentation) to optimise for certain situations. Example - the
new(ish) BIO_META tag that only CFQ understands. I can change the
way XFS issues bios to use this tag to make CFQ behave the same way
it used to w.r.t. metadata I/O from XFS, but then the deadline and
AS will probably regress because they don't understand that tag and
still need the old optimisations that just got removed. Ditto for
prioritised bio dispatch - CFQ supports it but none of the others
do.
There's nothing wrong with adding BIO_META (for example) and other
hints in _principle_. You should be able to ignore it with no adverse
effects. If its not used by a filesystem (and there's nothing else
competing to use the same disk), I would hope to see the same
performance as other kernels which don't have it.
Right, but it's what we need to do to make use of that optimisation
that is the problem. For XFS, it needs to replace the current
BIO_SYNC hints we use (even for async I/O) to get metadata
dispatched quickly. i.e. CFQ looks at the sync flag first then the
meta flag. Hence to take advantage of it, we need to remove the
BIO_SYNC hints we currently use which will change the behaviour on
all other elevators as a side effect.
This is the optimisation problem I'm refering to - the BIO_SYNC
usage was done years ago to get metadata dispatched quickly because
that is what all the elevators did with sync I/O. Now to optimise
for CFQ we need to remove that BIO_SYNC optimisation which is still
valid for the other elevators....
If the elevators are being changed in such a way that old filesystem
code which doesn't use new hint bits is running significantly slower,
surely that's blatant elevator regression, and that's where the bugs
should be reported and fixed?
Sure, but in reality getting ppl to go through the pain of triage is
extremely rare because it only takes 10s to change elevators and
make the problem go away...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
On Mon, Aug 25, 2008 at 01:01:47PM +0100, Jamie Lokier wrote:
quoted
Dave Chinner wrote:
quoted
To keep on top of this, we keep adding new variations and types and
expect the filesystems to make best use of them (without
documentation) to optimise for certain situations. Example - the
new(ish) BIO_META tag that only CFQ understands. I can change the
way XFS issues bios to use this tag to make CFQ behave the same way
it used to w.r.t. metadata I/O from XFS, but then the deadline and
AS will probably regress because they don't understand that tag and
still need the old optimisations that just got removed. Ditto for
prioritised bio dispatch - CFQ supports it but none of the others
do.
There's nothing wrong with adding BIO_META (for example) and other
hints in _principle_. You should be able to ignore it with no adverse
effects. If its not used by a filesystem (and there's nothing else
competing to use the same disk), I would hope to see the same
performance as other kernels which don't have it.
Right, but it's what we need to do to make use of that optimisation
that is the problem. For XFS, it needs to replace the current
BIO_SYNC hints we use (even for async I/O) to get metadata
dispatched quickly. i.e. CFQ looks at the sync flag first then the
meta flag. Hence to take advantage of it, we need to remove the
BIO_SYNC hints we currently use which will change the behaviour on
all other elevators as a side effect.
This is the optimisation problem I'm refering to - the BIO_SYNC
usage was done years ago to get metadata dispatched quickly because
that is what all the elevators did with sync I/O. Now to optimise
for CFQ we need to remove that BIO_SYNC optimisation which is still
valid for the other elevators....
quoted
If the elevators are being changed in such a way that old filesystem
code which doesn't use new hint bits is running significantly slower,
surely that's blatant elevator regression, and that's where the bugs
should be reported and fixed?
Sure, but in reality getting ppl to go through the pain of triage is
extremely rare because it only takes 10s to change elevators and
make the problem go away...
it sounds as if the various flag definitions have been evolving, would it be worthwhile to sep back and try to get the various filesystem folks to brainstorm together on what types of hints they would _like_ to see supported?
it sounds like you are using 'sync' for things where you really should be saying 'metadata' (or 'journal contents'), it's happened to work well enough in the past, but it's forcing you to keep tweaking the filesystems. it may be better to try and define things from the filesystem point of view and let the elevators do the tweaking.
basicly I'm proposing a complete rethink of the filesyste <-> elevator interface.
David Lang
From: Dave Chinner <david@fromorbit.com> Date: 2008-08-27 01:20:18
On Mon, Aug 25, 2008 at 08:50:14PM -0700, david@lang.hm wrote:
it sounds as if the various flag definitions have been evolving, would it
be worthwhile to sep back and try to get the various filesystem folks to
brainstorm together on what types of hints they would _like_ to see
supported?
Three types:
1. immediate dispatch - merge first with adjacent requests
then dispatch
2. delayed dispatch - queue for a short while to allow
merging of requests from above
3. bulk data - queue and merge. dispatch is completely
controlled by the elevator
Basically most metadata and log writes would fall into category 2,
which every logbufs/2 log writes or every log force using a category
1 to prevent log I/O from being stalled too long by other I/O.
Data writes from the filesystem would appear as category 3 (read and write)
and are subject to the specific elevator scheduling. That is, things
like the CFQ ionice throttling would work on the bulk data queue,
but not the other queues that the filesystem is using for metadata.
Tagging the I/O as a sync I/O can still be done, but that only
affects category 3 scheduling - category 1 or 2 would do the same
thing whether sync or async....
it sounds like you are using 'sync' for things where you really should be
saying 'metadata' (or 'journal contents'), it's happened to work well
enough in the past, but it's forcing you to keep tweaking the
filesystems.
Right, because there was no 'metadata' tagging, and 'sync' happened
to do exactly what we needed on all elevators at the time.
it may be better to try and define things from the
filesystem point of view and let the elevators do the tweaking.
basicly I'm proposing a complete rethink of the filesyste <-> elevator
interface.
Yeah, I've been saying that for a while w.r.t. the filesystem/block
layer interfaces, esp. now with discard requests, data integrity,
device alignment information, barriers, etc being exposed by the
layers below the filesystem, but with no interface for filesystems
to be able to access that information...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
On Mon, Aug 25, 2008 at 08:50:14PM -0700, david@lang.hm wrote:
quoted
it sounds as if the various flag definitions have been evolving, would it
be worthwhile to sep back and try to get the various filesystem folks to
brainstorm together on what types of hints they would _like_ to see
supported?
Three types:
1. immediate dispatch - merge first with adjacent requests
then dispatch
2. delayed dispatch - queue for a short while to allow
merging of requests from above
3. bulk data - queue and merge. dispatch is completely
controlled by the elevator
does this list change if you consider the fact that there may be a raid array or some more complex structure for the block device instead of a simple single disk partition?
since I am suggesting re-thinking the filesystem <-> elevator interface, is there anything you need to have the elevator tell the filesystem? (I'm thinking that this may be the path for the filesystem to learn things about the block device that's under it, is it a raid array, a solid-state drive, etc)
David Lang
Basically most metadata and log writes would fall into category 2,
which every logbufs/2 log writes or every log force using a category
1 to prevent log I/O from being stalled too long by other I/O.
Data writes from the filesystem would appear as category 3 (read and write)
and are subject to the specific elevator scheduling. That is, things
like the CFQ ionice throttling would work on the bulk data queue,
but not the other queues that the filesystem is using for metadata.
Tagging the I/O as a sync I/O can still be done, but that only
affects category 3 scheduling - category 1 or 2 would do the same
thing whether sync or async....
quoted
it sounds like you are using 'sync' for things where you really should be
saying 'metadata' (or 'journal contents'), it's happened to work well
enough in the past, but it's forcing you to keep tweaking the
filesystems.
Right, because there was no 'metadata' tagging, and 'sync' happened
to do exactly what we needed on all elevators at the time.
quoted
it may be better to try and define things from the
filesystem point of view and let the elevators do the tweaking.
basicly I'm proposing a complete rethink of the filesyste <-> elevator
interface.
Yeah, I've been saying that for a while w.r.t. the filesystem/block
layer interfaces, esp. now with discard requests, data integrity,
device alignment information, barriers, etc being exposed by the
layers below the filesystem, but with no interface for filesystems
to be able to access that information...
Cheers,
Dave.
From: Dave Chinner <david@fromorbit.com> Date: 2008-08-28 01:09:03
On Wed, Aug 27, 2008 at 02:54:28PM -0700, david@lang.hm wrote:
On Wed, 27 Aug 2008, Dave Chinner wrote:
quoted
On Mon, Aug 25, 2008 at 08:50:14PM -0700, david@lang.hm wrote:
quoted
it sounds as if the various flag definitions have been evolving, would it
be worthwhile to sep back and try to get the various filesystem folks to
brainstorm together on what types of hints they would _like_ to see
supported?
Three types:
1. immediate dispatch - merge first with adjacent requests
then dispatch
2. delayed dispatch - queue for a short while to allow
merging of requests from above
3. bulk data - queue and merge. dispatch is completely
controlled by the elevator
does this list change if you consider the fact that there may be a raid
array or some more complex structure for the block device instead of a
simple single disk partition?
No. The whole point of immediate dispatch is that those I/Os are
extremely latency sensitive (i.e. whole fs can stall waiting or
them), so it doesn't matter what the end target is. The faster the
storage subsystem, the more important it is to dispatch those
I/Os immediately to keep the pipes filled...
since I am suggesting re-thinking the filesystem <-> elevator interface,
is there anything you need to have the elevator tell the filesystem? (I'm
thinking that this may be the path for the filesystem to learn things
about the block device that's under it, is it a raid array, a solid-state
drive, etc)
Not so much the elevator, but the block layer in general. That is:
- capability reporting
- barriers and type
- discard support
- integrity support
- maximum number of I/Os that can be in flight
before congestion occurs
- geometry of the underlying storage
- independent domains within the device (e.g. boundaries
of linear concatentations)
- stripe unit/width per domain
- optimal I/O size per domain
- latency characteristics per domain
- notifiers to indicate change of status due to device
hotplug back up to the filesystem
- barrier status change
- geometry changes due to on-line volume modification
(e.g. raid5/6 rebuild after adding a new disk,
added another disk to a linear concat, etc)
I'm sure there's more, but that's the list quickly off the top of
my head.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com