From: Kevin Hilman <hidden> Date: 2014-06-09 20:47:42
Nicolas Pitre [off-list ref] writes:
On Sun, 8 Jun 2014, Lorenzo Pieralisi wrote:
quoted
On Sun, Jun 08, 2014 at 12:53:34AM +0100, Olof Johansson wrote:
quoted
Lorenzo,
Since you're emailing from @arm.com, some of this is to the wider
recipient and maybe not directly to you:
I am glad to reply and take blame since this is a debate definitely worth
having.
Great. Because I would like to steer this debate a little towards the
genuine cause rather than sticking to some particular consequences.
quoted
Guys, do not get me wrong here. There are fixes that can be deemed
acceptable in an OS, there are fixes that can't. I just can't help thinking
that Nicolas' patch is a nasty hack (and I am far, really really far from
blaming him for that, because that's the only patch that can fix that
issue in the kernel), and he perfectly knows that.
You know what? The more I think about my patch, the more I consider
this should be the standard way of setting up things unconditionally on
_all_ platforms using MCPM. Why? Because that's the most coherent thing
to do!
I agree.
I really think the kernel should either be responsible for the CCI or it
should not at all. And conversely for the bootloader. Right now we
have an implicit requirement that the bootloader should turn on the CCI,
but only for cold boot, and only for the boot cluster, and not for CPU
resuming from idle, and what other case we haven't thought about yet.
And as noticed this requirement is not documented.
In addition to being a firmware minimalist like Nico, what I find most
objectional to the bootloader approach is that even with CCI enabled by
the firmware, since it's a runtime requirement (for low-power idle or
suspend), the kernel has to handle it anyways. So you end up with a
partial solution in the firwmare (for boot cluster only) *and* a full
solution in the kernel. This doesn't make any sense, expecially because
the kernel might then have to do things differently on cold boot
vs. low-power idle/suspend or differently on the boot cluster vs. other
clusters. From a maintenance PoV, this is a mess and could easily lead
to just as many SoC specific hacks that are different across platforms.
Stated more simply: If the kernel has to manage the resource at runtime
due to low-power idle/suspend. I don't see any reason why it shouldn't
manage it at cold boot time also.
Kevin
From: Lorenzo Pieralisi <hidden> Date: 2014-06-09 22:26:05
On Mon, Jun 09, 2014 at 09:47:42PM +0100, Kevin Hilman wrote:
Nicolas Pitre [off-list ref] writes:
quoted
On Sun, 8 Jun 2014, Lorenzo Pieralisi wrote:
quoted
On Sun, Jun 08, 2014 at 12:53:34AM +0100, Olof Johansson wrote:
quoted
Lorenzo,
Since you're emailing from @arm.com, some of this is to the wider
recipient and maybe not directly to you:
I am glad to reply and take blame since this is a debate definitely worth
having.
Great. Because I would like to steer this debate a little towards the
genuine cause rather than sticking to some particular consequences.
I commented on Nico's patch because I did not like how it was
implemented (at least remove the CPU PM notifier calls please, because
they are not needed). I also said that's the only thing he could do,
and I still think that's not a nice way to use the cpu_suspend API
for something it was not designed for, that's what I wanted to say,
period.
quoted
quoted
Guys, do not get me wrong here. There are fixes that can be deemed
acceptable in an OS, there are fixes that can't. I just can't help thinking
that Nicolas' patch is a nasty hack (and I am far, really really far from
blaming him for that, because that's the only patch that can fix that
issue in the kernel), and he perfectly knows that.
You know what? The more I think about my patch, the more I consider
this should be the standard way of setting up things unconditionally on
_all_ platforms using MCPM. Why? Because that's the most coherent thing
to do!
I agree.
quoted
I really think the kernel should either be responsible for the CCI or it
should not at all. And conversely for the bootloader. Right now we
have an implicit requirement that the bootloader should turn on the CCI,
but only for cold boot, and only for the boot cluster, and not for CPU
resuming from idle, and what other case we haven't thought about yet.
And as noticed this requirement is not documented.
In addition to being a firmware minimalist like Nico, what I find most
objectional to the bootloader approach is that even with CCI enabled by
the firmware, since it's a runtime requirement (for low-power idle or
suspend), the kernel has to handle it anyways. So you end up with a
partial solution in the firwmare (for boot cluster only) *and* a full
solution in the kernel. This doesn't make any sense, expecially because
the kernel might then have to do things differently on cold boot
vs. low-power idle/suspend or differently on the boot cluster vs. other
clusters. From a maintenance PoV, this is a mess and could easily lead
to just as many SoC specific hacks that are different across platforms.
Stated more simply: If the kernel has to manage the resource at runtime
due to low-power idle/suspend. I don't see any reason why it shouldn't
manage it at cold boot time also.
If I am allowed to say something, here is a couple of thoughts.
1) CCI snoops and DVM enablement are secure operations, to do them in
non-secure world this must be overriden in firmware. You can argue,
you can think whatever you want, that's a fact. So, to use this
code SMP bit in SCTLR and CCI enablement must become non-secure
operations. This is a boot requirement for MCPM, right or wrong
it is up to platform designers to judge. If CCI and SMP enablement
are secure operations, we should not start adding random SMC calls
in the kernel, since managing coherency in the kernel would become
problematic, with lots of platform quirks. We do not want that to
happen, and I think we all agree on this.
2) (1) must be documented.
3) When I talked about consequences for CPUidle (implicit), I was referring
to all sort of hacks we had to come up to bring devices like SPC
(remember ? I remember very very well unfortunately for me), or whatever
power controller up in the kernel early, too early to fit in any
existing kernel device framework. There is still no solution to that, and
the only way that code can exist is in mach- code. Right or wrong,
that's a second fact and in my opinion that's not nice for the ARM
kernel.
4) When I am talking about firmware I am talking about sequences that
are very close to HW (disabling C bit, cleaning caches, exiting
coherency). Erratas notwithstanding, they are being standardized at
ARM the best we can. They might even end up being implemented in HW
in the not so far future. I understand they are tricky, I understand
they take lots of time to implement them and to debug them, what I
want to say is that they are becoming standard and we _must_ reuse the
same code for all ARM platforms. You can implement them in MCPM (see
(1)) or in firmware (and please do not start painting me as firmware
hugger here, I am referring to standard power down sequences that
again, are very close to HW state machines and more importantly if they
HAVE to run in secure world that's the only solution we have unless you
want to split race conditions between kernel and secure world).
5) I agree that the CCI enablement in TC2 (bootmon for cold boot and
kernel for warm-boot is wrong, nothing to say and it was not the
reason I commented on Nico's patch - I think I explained to you
thoroughly why now).
That's what I had to say, I hope it helps.
Thanks,
Lorenzo
From: Nicolas Pitre <hidden> Date: 2014-06-10 04:25:47
On Mon, 9 Jun 2014, Lorenzo Pieralisi wrote:
I commented on Nico's patch because I did not like how it was
implemented (at least remove the CPU PM notifier calls please, because
they are not needed).
OK no problem. That's easy enough. I added them to play it safe as a
test patch in case some VFP content could be lost somehow by looping
back through the CPU init code for example, and needed to be saved.
I also said that's the only thing he could do, and I still think
that's not a nice way to use the cpu_suspend API for something it was
not designed for, that's what I wanted to say, period.
Well... Maybe it wasn't designed for that, but it certainly can be used
for that. And with no modifications to the core code, making this
solution fairly elegant. This is not so different from, say, the BPF
code being reused for seccomp_filters. BPF wasn't designed for system
call filtering, but it happens to work well.
If I am allowed to say something, here is a couple of thoughts.
1) CCI snoops and DVM enablement are secure operations, to do them in
non-secure world this must be overriden in firmware. You can argue,
you can think whatever you want, that's a fact. So, to use this
code SMP bit in SCTLR and CCI enablement must become non-secure
operations. This is a boot requirement for MCPM, right or wrong
it is up to platform designers to judge. If CCI and SMP enablement
are secure operations, we should not start adding random SMC calls
in the kernel, since managing coherency in the kernel would become
problematic, with lots of platform quirks. We do not want that to
happen, and I think we all agree on this.
One could certainly question the need for so many controls handled in
secure world. But that is not the point.
Here we're talking about MCPM. That implies the kernel has control over
SCTLR.SMP and the CCI. If those things aren't under the kernel's
control, then MCPM is of no use to you.
Therefore, if you do want to use MCPM, then this implies the kernel has
access to the CCI. And if it has access to it, then it should turn it on
by itself in all cases to be consistent, not only in half of the cases.
2) (1) must be documented.
Absolutely. But let's be coherent in the implementation so the
documentation is as simple as it can be.
3) When I talked about consequences for CPUidle (implicit), I was referring
to all sort of hacks we had to come up to bring devices like SPC
(remember ? I remember very very well unfortunately for me), or whatever
power controller up in the kernel early, too early to fit in any
existing kernel device framework. There is still no solution to that, and
the only way that code can exist is in mach- code. Right or wrong,
that's a second fact and in my opinion that's not nice for the ARM
kernel.
I disagree. This can perfectly be turned into driver code. If we need
it too early for existing kernel device framework to handle this
properly, then the solution is to extend the existing framework or
create another one specially for that purpose. This may not be obvious
when TC2 is the first/only platform in that situation, but if more
platforms have the same need then it'll be easier to abstract
commonalities into a framework.
Saying that no framework exists today or/and upstream maintainers are
being difficult is _not_ a reason for throwing your hands up and e.g.
shoving all this code into firmware instead.
4) When I am talking about firmware I am talking about sequences that
are very close to HW (disabling C bit, cleaning caches, exiting
coherency). Erratas notwithstanding, they are being standardized at
ARM the best we can. They might even end up being implemented in HW
in the not so far future. I understand they are tricky, I understand
they take lots of time to implement them and to debug them, what I
want to say is that they are becoming standard and we _must_ reuse the
same code for all ARM platforms. You can implement them in MCPM (see
(1)) or in firmware (and please do not start painting me as firmware
hugger here, I am referring to standard power down sequences that
again, are very close to HW state machines
That's where the disconnect lies. On the one hand you say "I understand
they are tricky, I understand they take lots of time to implement them
and to debug them" and on the other hand you say "They might end up being
implemented in HW in the not so far future." That simply makes no
economical sense at all!
When some operation is 1) tricky and takes time to debug, and 2) not
performance critical (no one is trying to get in and out of idle or
hibernation a billion times per second), then you should never ever put
such a thing in firmware, and hardware should be completely out of the
question!
and more importantly if they
HAVE to run in secure world that's the only solution we have unless you
want to split race conditions between kernel and secure world).
If they HAVE to run in secure world then your secure world architecture
is simply misdesigned, period. Someone must have ignored the economics
of modern software development to have come up with this.
5) I agree that the CCI enablement in TC2 (bootmon for cold boot and
kernel for warm-boot is wrong, nothing to say and it was not the
reason I commented on Nico's patch - I think I explained to you
thoroughly why now).
OK. Let's start by agreeing on the spirit behind my patch then. The
actual patch implementation details are a secondary concern and open for
discussion.
Nicolas
From: Lorenzo Pieralisi <hidden> Date: 2014-06-10 09:19:38
On Tue, Jun 10, 2014 at 05:25:47AM +0100, Nicolas Pitre wrote:
On Mon, 9 Jun 2014, Lorenzo Pieralisi wrote:
quoted
I commented on Nico's patch because I did not like how it was
implemented (at least remove the CPU PM notifier calls please, because
they are not needed).
OK no problem. That's easy enough. I added them to play it safe as a
test patch in case some VFP content could be lost somehow by looping
back through the CPU init code for example, and needed to be saved.
Ok, thanks.
quoted
I also said that's the only thing he could do, and I still think
that's not a nice way to use the cpu_suspend API for something it was
not designed for, that's what I wanted to say, period.
Well... Maybe it wasn't designed for that, but it certainly can be used
for that. And with no modifications to the core code, making this
solution fairly elegant. This is not so different from, say, the BPF
code being reused for seccomp_filters. BPF wasn't designed for system
call filtering, but it happens to work well.
You defined yourself "not a small thing", you know what you are doing
and that's enough for me. Please respect that when I reviewed it I thought
that was a hack. cpu_suspend is being used for many things in the kernel
and consolidating that took a while, please comment the code, that's all I
am asking you.
quoted
If I am allowed to say something, here is a couple of thoughts.
1) CCI snoops and DVM enablement are secure operations, to do them in
non-secure world this must be overriden in firmware. You can argue,
you can think whatever you want, that's a fact. So, to use this
code SMP bit in SCTLR and CCI enablement must become non-secure
operations. This is a boot requirement for MCPM, right or wrong
it is up to platform designers to judge. If CCI and SMP enablement
are secure operations, we should not start adding random SMC calls
in the kernel, since managing coherency in the kernel would become
problematic, with lots of platform quirks. We do not want that to
happen, and I think we all agree on this.
One could certainly question the need for so many controls handled in
secure world. But that is not the point.
Here we're talking about MCPM. That implies the kernel has control over
SCTLR.SMP and the CCI. If those things aren't under the kernel's
control, then MCPM is of no use to you.
ACTLR.SMP for the sake of precision and it was my typo, sorry. That's
all I wanted to read, so nothing to add.
Therefore, if you do want to use MCPM, then this implies the kernel has
access to the CCI. And if it has access to it, then it should turn it on
by itself in all cases to be consistent, not only in half of the cases.
I agree.
quoted
2) (1) must be documented.
Absolutely. But let's be coherent in the implementation so the
documentation is as simple as it can be.
Ditto.
quoted
3) When I talked about consequences for CPUidle (implicit), I was referring
to all sort of hacks we had to come up to bring devices like SPC
(remember ? I remember very very well unfortunately for me), or whatever
power controller up in the kernel early, too early to fit in any
existing kernel device framework. There is still no solution to that, and
the only way that code can exist is in mach- code. Right or wrong,
that's a second fact and in my opinion that's not nice for the ARM
kernel.
I disagree. This can perfectly be turned into driver code. If we need
it too early for existing kernel device framework to handle this
properly, then the solution is to extend the existing framework or
create another one specially for that purpose. This may not be obvious
when TC2 is the first/only platform in that situation, but if more
platforms have the same need then it'll be easier to abstract
commonalities into a framework.
Saying that no framework exists today or/and upstream maintainers are
being difficult is _not_ a reason for throwing your hands up and e.g.
shoving all this code into firmware instead.
You have a point, as long as we are all aware and we do not forget this
is a major problem, not a minor one. I do want to see a consolidate
story for CPUidle for ARM and this bullet is definitely part of the
picture. On a side note, you made me smile, it sounded like I wanted
to bury SPC code in firmware or anywhere else as long as it is not in the
kernel, which in a way is a true statement since I abhor that code =)
quoted
4) When I am talking about firmware I am talking about sequences that
are very close to HW (disabling C bit, cleaning caches, exiting
coherency). Erratas notwithstanding, they are being standardized at
ARM the best we can. They might even end up being implemented in HW
in the not so far future. I understand they are tricky, I understand
they take lots of time to implement them and to debug them, what I
want to say is that they are becoming standard and we _must_ reuse the
same code for all ARM platforms. You can implement them in MCPM (see
(1)) or in firmware (and please do not start painting me as firmware
hugger here, I am referring to standard power down sequences that
again, are very close to HW state machines
That's where the disconnect lies. On the one hand you say "I understand
they are tricky, I understand they take lots of time to implement them
and to debug them" and on the other hand you say "They might end up being
implemented in HW in the not so far future." That simply makes no
economical sense at all!
I wanted to say Nico that those operations are so intrinsic for all ARM
cores you can almost think of them as part of the ISA and certainly
HW knows it better than SW when a processor has nothing to do, it does
idle core units all the time without software interaction, going power off
(on cue) could just be one step further and solve those pesky races in HW.
When some operation is 1) tricky and takes time to debug, and 2) not
performance critical (no one is trying to get in and out of idle or
hibernation a billion times per second), then you should never ever put
such a thing in firmware, and hardware should be completely out of the
question!
We can debate this offline, it is an interesting topic in particular on
the performance critical side of things; as for firmware see my point
on security, as I already mentioned.
quoted
and more importantly if they
HAVE to run in secure world that's the only solution we have unless you
want to split race conditions between kernel and secure world).
If they HAVE to run in secure world then your secure world architecture
is simply misdesigned, period. Someone must have ignored the economics
of modern software development to have come up with this.
That's your opinion and I respect that. What I said, and that's a fact
not an opinion, is that if those operations are required to be secure in a
platform, there is not much you can do apart from preventing races where the
race conditions are, ie in secure world.
quoted
5) I agree that the CCI enablement in TC2 (bootmon for cold boot and
kernel for warm-boot is wrong, nothing to say and it was not the
reason I commented on Nico's patch - I think I explained to you
thoroughly why now).
OK. Let's start by agreeing on the spirit behind my patch then. The
actual patch implementation details are a secondary concern and open for
discussion.
I agree on the spirit of the patch, my concern was about its implementation.
Thank you, this was a constructive discussion.
Lorenzo
Hi Nico,
Sorry, I can't stay away from this thread ;)
On Tue, Jun 10, 2014 at 12:25:47AM -0400, Nicolas Pitre wrote:
On Mon, 9 Jun 2014, Lorenzo Pieralisi wrote:
quoted
4) When I am talking about firmware I am talking about sequences that
are very close to HW (disabling C bit, cleaning caches, exiting
coherency). Erratas notwithstanding, they are being standardized at
ARM the best we can. They might even end up being implemented in HW
in the not so far future. I understand they are tricky, I understand
they take lots of time to implement them and to debug them, what I
want to say is that they are becoming standard and we _must_ reuse the
same code for all ARM platforms. You can implement them in MCPM (see
(1)) or in firmware (and please do not start painting me as firmware
hugger here, I am referring to standard power down sequences that
again, are very close to HW state machines
That's where the disconnect lies. On the one hand you say "I understand
they are tricky, I understand they take lots of time to implement them
and to debug them" and on the other hand you say "They might end up being
implemented in HW in the not so far future." That simply makes no
economical sense at all!
It makes lots of sense, though not from a software maintainability
perspective. It would be nice if everything still looked like ARM7TDMI
but in the race for performance (vs power), hardware becomes more
complex and it's not just the CPU but adjacent parts like interconnects,
caches, asynchronous bridges, voltage shifters, memory controllers,
clocks/PLLs etc. Many of these are simply hidden from the high level OS
like Linux because the OS assumes certain configuration (e.g. access to
memory) and it's only the hardware itself that knows in what order they
can be turned on or off (when triggered explicitly by the OS or an
external event). Having an dedicated power controller (e.g. M-class
processor) to handle some of these is a rather flexible approach, other
bits require RTL (and usually impossible to update).
When some operation is 1) tricky and takes time to debug, and 2) not
performance critical (no one is trying to get in and out of idle or
hibernation a billion times per second), then you should never ever put
such a thing in firmware, and hardware should be completely out of the
question!
I agree that things can go wrong (both in hardware and software, no
matter where it runs) but please don't think that such power
architecture has been specifically engineered to hide the hardware from
Linux. It's a necessity for complex systems and the optimal solution is
not always simplification (it's not just ARM+vendors doing this, just
look at the power model of modern x86 processors, hidden nicely from the
software behind a few registers while making things harder for scheduler
which cannot rely on a constant performance level; but it's a trade-off
they are happy to make).
quoted
and more importantly if they
HAVE to run in secure world that's the only solution we have unless you
want to split race conditions between kernel and secure world).
If they HAVE to run in secure world then your secure world architecture
is simply misdesigned, period. Someone must have ignored the economics
of modern software development to have come up with this.
That's the trade-off between software complexity and hardware cost,
gates, power consumption. You can do proper physical separation of the
secure services but this would require a separate CPU that is rarely
used and adds to the overall SoC cost. On large scale hardware
deployment, it's exactly economics that matter and these translate into
hardware cost. The software cost is irrelevant here, whether we like it
or not.
--
Catalin
From: Nicolas Pitre <hidden> Date: 2014-06-10 16:49:01
On Tue, 10 Jun 2014, Catalin Marinas wrote:
Hi Nico,
Sorry, I can't stay away from this thread ;)
;-)
On Tue, Jun 10, 2014 at 12:25:47AM -0400, Nicolas Pitre wrote:
quoted
On Mon, 9 Jun 2014, Lorenzo Pieralisi wrote:
quoted
4) When I am talking about firmware I am talking about sequences that
are very close to HW (disabling C bit, cleaning caches, exiting
coherency). Erratas notwithstanding, they are being standardized at
ARM the best we can. They might even end up being implemented in HW
in the not so far future. I understand they are tricky, I understand
they take lots of time to implement them and to debug them, what I
want to say is that they are becoming standard and we _must_ reuse the
same code for all ARM platforms. You can implement them in MCPM (see
(1)) or in firmware (and please do not start painting me as firmware
hugger here, I am referring to standard power down sequences that
again, are very close to HW state machines
That's where the disconnect lies. On the one hand you say "I understand
they are tricky, I understand they take lots of time to implement them
and to debug them" and on the other hand you say "They might end up being
implemented in HW in the not so far future." That simply makes no
economical sense at all!
It makes lots of sense, though not from a software maintainability
perspective. It would be nice if everything still looked like ARM7TDMI
but in the race for performance (vs power), hardware becomes more
complex and it's not just the CPU but adjacent parts like interconnects,
caches, asynchronous bridges, voltage shifters, memory controllers,
clocks/PLLs etc. Many of these are simply hidden from the high level OS
like Linux because the OS assumes certain configuration (e.g. access to
memory) and it's only the hardware itself that knows in what order they
can be turned on or off (when triggered explicitly by the OS or an
external event).
I agree when the hardware has to handle parallel dependencies ordered in
waterfall style. In such cases there is usually no point relying on
software to implement what is nevertheless simple determinism with
no possible alternative usage.
But the *most* important thing is what you put in parents, so let me
emphasize on what you just said:
When triggered _explicitly_ by the OS or external events
Having an dedicated power controller (e.g. M-class
processor) to handle some of these is a rather flexible approach, other
bits require RTL (and usually impossible to update).
The M-class processor should be treated the same way as firmware. It
ought to be flexible (certainly more than hardwired hardware), but it
shares all the same downsides as firmware and the same concerns apply.
quoted
When some operation is 1) tricky and takes time to debug, and 2) not
performance critical (no one is trying to get in and out of idle or
hibernation a billion times per second), then you should never ever put
such a thing in firmware, and hardware should be completely out of the
question!
I agree that things can go wrong (both in hardware and software, no
matter where it runs) but please don't think that such power
architecture has been specifically engineered to hide the hardware from
Linux. It's a necessity for complex systems and the optimal solution is
not always simplification (it's not just ARM+vendors doing this, just
look at the power model of modern x86 processors, hidden nicely from the
software behind a few registers while making things harder for scheduler
which cannot rely on a constant performance level; but it's a trade-off
they are happy to make).
I'll claim that this is a bad tradeoff. And the reason why some
hardware architects might think it is a good one is because so far we
really sucked at software based power management in Linux (and possibly
other OSes as well). Hence the (fairly recent) realization that power
management has to be integrated and under control of the scheduler
rather than existing as some ad hoc subsystem.
The reaction from the hardware people often is "the software is crap and
makes our hardware look bad, we know better, so let's make it easier on
those poor software dudes by handling power management in hardware
instead". But ultimately the hardware just can't predict things like
software can. It might do a better job than the current software state
of affairs, but most likely not be as efficient as a proper software
architecture. The hardware may only be reactive, whereas the software
can be proactive (when properly done that is).
I sense from your paragraph above that ARM might be going the same
direction as X86 and that would be very sad. Maybe the best compromise
would be for all knobs to be made available to software if software
wants to turn off the hardware auto-pilot and take control. This way
the hardware guys would cover their arses while still allowing for the
possibility that software might be able to out perform the hardware
solution.
quoted
quoted
and more importantly if they
HAVE to run in secure world that's the only solution we have unless you
want to split race conditions between kernel and secure world).
If they HAVE to run in secure world then your secure world architecture
is simply misdesigned, period. Someone must have ignored the economics
of modern software development to have come up with this.
That's the trade-off between software complexity and hardware cost,
gates, power consumption. You can do proper physical separation of the
secure services but this would require a separate CPU that is rarely
used and adds to the overall SoC cost. On large scale hardware
deployment, it's exactly economics that matter and these translate into
hardware cost. The software cost is irrelevant here, whether we like it
or not.
I agree with you on the hardware cost (and the same argument applies to
power management by the way). But once the hardware is there, the
software cost has to be optimized the same way.
From a cost perspective, firmware is always a magnitude more costly to
develop and to fix and maintain afterwards than kernel code. So,
without requiring full physical separation increasing the hardware cost,
I think the software architecture would benefit from a rethought,
possibly with the help of small and cheap hardware enhancements. I
really think not enough attention has been dedicated to that aspect.
Nicolas
On Tue, Jun 10, 2014 at 05:49:01PM +0100, Nicolas Pitre wrote:
On Tue, 10 Jun 2014, Catalin Marinas wrote:
quoted
On Tue, Jun 10, 2014 at 12:25:47AM -0400, Nicolas Pitre wrote:
quoted
On Mon, 9 Jun 2014, Lorenzo Pieralisi wrote:
quoted
4) When I am talking about firmware I am talking about sequences that
are very close to HW (disabling C bit, cleaning caches, exiting
coherency). Erratas notwithstanding, they are being standardized at
ARM the best we can. They might even end up being implemented in HW
in the not so far future. I understand they are tricky, I understand
they take lots of time to implement them and to debug them, what I
want to say is that they are becoming standard and we _must_ reuse the
same code for all ARM platforms. You can implement them in MCPM (see
(1)) or in firmware (and please do not start painting me as firmware
hugger here, I am referring to standard power down sequences that
again, are very close to HW state machines
That's where the disconnect lies. On the one hand you say "I understand
they are tricky, I understand they take lots of time to implement them
and to debug them" and on the other hand you say "They might end up being
implemented in HW in the not so far future." That simply makes no
economical sense at all!
It makes lots of sense, though not from a software maintainability
perspective. It would be nice if everything still looked like ARM7TDMI
but in the race for performance (vs power), hardware becomes more
complex and it's not just the CPU but adjacent parts like interconnects,
caches, asynchronous bridges, voltage shifters, memory controllers,
clocks/PLLs etc. Many of these are simply hidden from the high level OS
like Linux because the OS assumes certain configuration (e.g. access to
memory) and it's only the hardware itself that knows in what order they
can be turned on or off (when triggered explicitly by the OS or an
external event).
I agree when the hardware has to handle parallel dependencies ordered in
waterfall style. In such cases there is usually no point relying on
software to implement what is nevertheless simple determinism with
no possible alternative usage.
But the *most* important thing is what you put in parens, so let me
emphasize on what you just said:
When triggered _explicitly_ by the OS or external events
I don't think anyone is arguing that the policy should not be in the OS.
But part of the mechanism can be in the OS and part in firmware, SCP or
hardware. The kernel part can be a simple PSCI call or more complex
setup (possibly MCPM-based) which usually ends up with a WFI. This WFI,
however, triggers further hardware changes which may be handled by
dedicated power controller.
quoted
Having an dedicated power controller (e.g. M-class
processor) to handle some of these is a rather flexible approach, other
bits require RTL (and usually impossible to update).
The M-class processor should be treated the same way as firmware. It
ought to be flexible (certainly more than hardwired hardware), but it
shares all the same downsides as firmware and the same concerns apply.
Yes, we can treat it as firmware, but we don't have a better alternative
to move the functionality into the kernel (well, we could at least allow
the kernel to load a binary blob and restart the controller).
quoted
quoted
When some operation is 1) tricky and takes time to debug, and 2) not
performance critical (no one is trying to get in and out of idle or
hibernation a billion times per second), then you should never ever put
such a thing in firmware, and hardware should be completely out of the
question!
I agree that things can go wrong (both in hardware and software, no
matter where it runs) but please don't think that such power
architecture has been specifically engineered to hide the hardware from
Linux. It's a necessity for complex systems and the optimal solution is
not always simplification (it's not just ARM+vendors doing this, just
look at the power model of modern x86 processors, hidden nicely from the
software behind a few registers while making things harder for scheduler
which cannot rely on a constant performance level; but it's a trade-off
they are happy to make).
I'll claim that this is a bad tradeoff. And the reason why some
hardware architects might think it is a good one is because so far we
really sucked at software based power management in Linux (and possibly
other OSes as well). Hence the (fairly recent) realization that power
management has to be integrated and under control of the scheduler
rather than existing as some ad hoc subsystem.
But even this is a complex problem. Anyway, I don't think the (ARM at
least) hardware guys aim to take over the cpufreq or Linux scheduler
functionality. Their concern is rather mechanism rather than policy.
The reaction from the hardware people often is "the software is crap and
makes our hardware look bad, we know better, so let's make it easier on
those poor software dudes by handling power management in hardware
instead". But ultimately the hardware just can't predict things like
software can. It might do a better job than the current software state
of affairs, but most likely not be as efficient as a proper software
architecture. The hardware may only be reactive, whereas the software
can be proactive (when properly done that is).
Indeed. But that's not the aim of the power controller on our boards.
It's just a mechanism for safely changing sleep states, CPU frequencies
but entirely under the OS decision.
I sense from your paragraph above that ARM might be going the same
direction as X86 and that would be very sad. Maybe the best compromise
would be for all knobs to be made available to software if software
wants to turn off the hardware auto-pilot and take control. This way
the hardware guys would cover their arses while still allowing for the
possibility that software might be able to out perform the hardware
solution.
I'm definitely not suggesting ARM is going the same route. Just trying
to show that ARM is slightly better here.
As a personal opinion, I like the simplicity of writing a register to
change the P-state but I don't like the non-determinism of the x86
hardware w.r.t. CPU performance. There are however some "policy" aspects
which I find interesting (like detecting whether the workload is memory
bound and automatically lowering the CPU frequency; the OS cannot react
this fast).
quoted
quoted
quoted
and more importantly if they
HAVE to run in secure world that's the only solution we have unless you
want to split race conditions between kernel and secure world).
If they HAVE to run in secure world then your secure world architecture
is simply misdesigned, period. Someone must have ignored the economics
of modern software development to have come up with this.
That's the trade-off between software complexity and hardware cost,
gates, power consumption. You can do proper physical separation of the
secure services but this would require a separate CPU that is rarely
used and adds to the overall SoC cost. On large scale hardware
deployment, it's exactly economics that matter and these translate into
hardware cost. The software cost is irrelevant here, whether we like it
or not.
I agree with you on the hardware cost (and the same argument applies to
power management by the way). But once the hardware is there, the
software cost has to be optimized the same way.
From a cost perspective, firmware is always a magnitude more costly to
develop and to fix and maintain afterwards than kernel code. So,
without requiring full physical separation increasing the hardware cost,
I think the software architecture would benefit from a rethought,
possibly with the help of small and cheap hardware enhancements. I
really think not enough attention has been dedicated to that aspect.
I fully agree (and I think Linaro is well positioned for this ;),
possibly as an extension of the boot+firmware architecture).
--
Catalin
From: Nicolas Pitre <hidden> Date: 2014-06-10 19:15:53
On Tue, 10 Jun 2014, Catalin Marinas wrote:
On Tue, Jun 10, 2014 at 05:49:01PM +0100, Nicolas Pitre wrote:
quoted
The M-class processor should be treated the same way as firmware. It
ought to be flexible (certainly more than hardwired hardware), but it
shares all the same downsides as firmware and the same concerns apply.
Yes, we can treat it as firmware, but we don't have a better alternative
to move the functionality into the kernel (well, we could at least allow
the kernel to load a binary blob and restart the controller).
That would address the "easy to update in the field" side of the story.
So far I've not seen this aspect been addressed with a serious plan
anywhere.
quoted
The reaction from the hardware people often is "the software is crap and
makes our hardware look bad, we know better, so let's make it easier on
those poor software dudes by handling power management in hardware
instead". But ultimately the hardware just can't predict things like
software can. It might do a better job than the current software state
of affairs, but most likely not be as efficient as a proper software
architecture. The hardware may only be reactive, whereas the software
can be proactive (when properly done that is).
Indeed. But that's not the aim of the power controller on our boards.
It's just a mechanism for safely changing sleep states, CPU frequencies
but entirely under the OS decision.
Sure. But then you might want to consider that some usage scenarios
might benefit from the ability to abort a request, or monitor the
progress of a request for software timing purposes, or accept parallel
requests rather than serialize them, etc. Given the flexibility to
extend beyond a rigid interface, the system may become even more
efficient overall, albeit with added complexity in the implementation.
But for that to work it has to be cheaply achievable.
quoted
I sense from your paragraph above that ARM might be going the same
direction as X86 and that would be very sad. Maybe the best compromise
would be for all knobs to be made available to software if software
wants to turn off the hardware auto-pilot and take control. This way
the hardware guys would cover their arses while still allowing for the
possibility that software might be able to out perform the hardware
solution.
I'm definitely not suggesting ARM is going the same route. Just trying
to show that ARM is slightly better here.
As a personal opinion, I like the simplicity of writing a register to
change the P-state but I don't like the non-determinism of the x86
hardware w.r.t. CPU performance. There are however some "policy" aspects
which I find interesting (like detecting whether the workload is memory
bound and automatically lowering the CPU frequency; the OS cannot react
this fast).
This is not really a policy. This is a straight-forward waterfall
dependency. There is simply nothing you can do with those CPU clock
cycles when stalled most of the time waiting for memory queries to come
back so the choice is obvious. If however this has a significant impact
on code execution speed then this becomes a tradeoff and the choice to
use this feature or not (the policy) must be implemented in software.
Nicolas