From: Anton Blanchard <hidden> Date: 2015-08-05 04:03:01
Hi,
While looking at traces of kernel workloads, I noticed places where gcc
used a large number of non volatiles. Some of these functions
did very little work, and we spent most of our time saving the
non volatiles to the stack and reading them back.
It made me wonder if we have the right ratio of volatile to non
volatile GPRs. Since the kernel is completely self contained, we could
potentially change that ratio.
Attached is a quick hack to gcc and the kernel to decrease the number
of non volatile GPRs to 8. I'm not sure if this is a good idea (and if
the volatile to non volatile ratio is right), but this gives us
something to play with.
Anton
Hi Anton,
On Wed, Aug 05, 2015 at 02:03:00PM +1000, Anton Blanchard wrote:
While looking at traces of kernel workloads, I noticed places where gcc
used a large number of non volatiles. Some of these functions
did very little work, and we spent most of our time saving the
non volatiles to the stack and reading them back.
That is something that should be fixed in GCC -- do you have an example
of such a function?
It made me wonder if we have the right ratio of volatile to non
volatile GPRs. Since the kernel is completely self contained, we could
potentially change that ratio.
Attached is a quick hack to gcc and the kernel to decrease the number
of non volatile GPRs to 8. I'm not sure if this is a good idea (and if
the volatile to non volatile ratio is right), but this gives us
something to play with.
Instead of the GCC hack you can add a bunch of -fcall-used-r14 etc.
options; does that not work for you?
Segher
From: Bill Schmidt <hidden> Date: 2015-08-07 05:55:46
I agree with Segher. We already know we have opportunities to do a better
job with shrink-wrapping (pushing this kind of useless activity down past
early exits), so having examples of code to look at to improve this would
be useful.
-- Bill
Bill Schmidt, Ph.D.
Linux on Power Toolchain
IBM Linux Technology Center
wschmidt@us.ibm.com (507) 319-6873
From: Segher Boessenkool <redacted>
To: Anton Blanchard <redacted>
Cc: linuxppc-dev@lists.ozlabs.org, Michael
Gschwind/Watson/IBM@IBMUS, Alan Modra [off-list ref], Bill
Schmidt/Rochester/IBM@IBMUS, Ulrich Weigand
[off-list ref], paulus@samba.org
Date: 08/05/2015 06:20 AM
Subject: Re: RFC: Reducing the number of non volatile GPRs in the ppc64
kernel
Hi Anton,
On Wed, Aug 05, 2015 at 02:03:00PM +1000, Anton Blanchard wrote:
While looking at traces of kernel workloads, I noticed places where gcc
used a large number of non volatiles. Some of these functions
did very little work, and we spent most of our time saving the
non volatiles to the stack and reading them back.
That is something that should be fixed in GCC -- do you have an example
of such a function?
It made me wonder if we have the right ratio of volatile to non
volatile GPRs. Since the kernel is completely self contained, we could
potentially change that ratio.
Attached is a quick hack to gcc and the kernel to decrease the number
of non volatile GPRs to 8. I'm not sure if this is a good idea (and if
the volatile to non volatile ratio is right), but this gives us
something to play with.
Instead of the GCC hack you can add a bunch of -fcall-used-r14 etc.
options; does that not work for you?
Segher
From: Anton Blanchard <hidden> Date: 2015-08-10 04:52:30
Hi Bill, Segher,
I agree with Segher. We already know we have opportunities to do a
better job with shrink-wrapping (pushing this kind of useless
activity down past early exits), so having examples of code to look
at to improve this would be useful.
On Mon, Aug 10, 2015 at 02:52:28PM +1000, Anton Blanchard wrote:
Hi Bill, Segher,
quoted
I agree with Segher. We already know we have opportunities to do a
better job with shrink-wrapping (pushing this kind of useless
activity down past early exits), so having examples of code to look
at to improve this would be useful.
I'll look out for specific examples. I noticed this one today when
analysing malloc(8). It is an instruction trace of _int_malloc().
The overall function is pretty huge, which I assume leads to gcc using
so many non volatiles.
That is one part of it; also GCC deals out volatiles too generously.
Perhaps in this case we should separate out the
slow path into another function marked noinline.
Or GCC could do that, effectively at least.
This is just an upstream glibc build, but I'll send the preprocessed
source off list.
Thanks :-)
[snip code]
After the prologue there are 46 insns executed before the epilogue.
Many of those are conditional branches (that are not executed); it is
all fall-through until it jumps to the "tail" (the few insns before
the epilogue). GCC knows how to duplicate a tail so that it can do
shrink-wrapping (the original tail needs to be followed by an epilogue,
the duplicated one does not want one); but it can only do it in very
simple cases (one basic block or at least no control flow), and that
is not the case here. We need to handle more generic tails.
This seems related to (if not the same as!) <http://gcc.gnu.org/PR51982>.
Segher
On Tue, Aug 11, 2015 at 03:08:29PM -0500, Segher Boessenkool wrote:
[snip code]
After the prologue there are 46 insns executed before the epilogue.
Many of those are conditional branches (that are not executed); it is
all fall-through until it jumps to the "tail" (the few insns before
the epilogue). GCC knows how to duplicate a tail so that it can do
shrink-wrapping (the original tail needs to be followed by an epilogue,
the duplicated one does not want one); but it can only do it in very
simple cases (one basic block or at least no control flow), and that
is not the case here. We need to handle more generic tails.
And never mind the elephant in the room: the "fastpath" instructions
already use a few non-volatile registers, and the shrink-wrap pass
(which runs after register allocation) cannot fix that. Ugh.
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2015-08-14 02:01:10
On Wed, 2015-08-05 at 14:03 +1000, Anton Blanchard wrote:
Hi,
While looking at traces of kernel workloads, I noticed places where gcc
used a large number of non volatiles. Some of these functions
did very little work, and we spent most of our time saving the
non volatiles to the stack and reading them back.
It made me wonder if we have the right ratio of volatile to non
volatile GPRs. Since the kernel is completely self contained, we could
potentially change that ratio.
Attached is a quick hack to gcc and the kernel to decrease the number
of non volatile GPRs to 8. I'm not sure if this is a good idea (and if
the volatile to non volatile ratio is right), but this gives us
something to play with.
OK, interesting idea. Can't say I'd ever though of that.
I'm thinking we'd want some pretty solid analysis of the resulting code-gen and
real world perf before we made a switch like that.
Presumably it's going to hurt our null syscall, due to the added save/restores,
but hopefully help with paths that do actual work.
If the caller is actually using the non-volatiles then presumably it will be a
wash, because the caller will have to do the save anyway. Though maybe it would
still be a win because the caller can do the saves & restores when it needs to
rather than all in a block.
I'm also not clear on how it would affect folks who build modules separate from
the kernel. We'd have to make sure they had the right GCC, or things would go
badly wrong, unless it can be done with command line flags? I don't know how
much we care about that but distros presumably do.
cheers