From: David Edelsohn <hidden> Date: 2001-02-08 22:08:32
quoted
quoted
quoted
quoted
Cort Dougan writes:
} that the port is not using the PowerPC architecture as intended. By not
} utilizing the hardware assists, the port is not performing at its optimal
} level.
Cort> I have data, and have written a paper with Victor and Paul, showing that we
Cort> get performance _increases_ by not using the PowerPC MMU architecture as
Cort> intended. I think the PPC architecture intentions for the hash table and
Cort> TLB and very very poor and restrictive. The 603 was a good step forward
Cort> but the 750, 7400 and follow-ons have been steps backwards from this good
Cort> start.
Your paper was a very novel and good solution to a performance
problem that you detected in the VMM design. I already mentioned one of
the problems with the VMM design causing double misses on write faults.
Let me reference the reasoning that Orran Krieger, Marc Auslander, and I
wrote to you about in March 1999 after Orran attended your talk at OSDI:
"Your paper discussess an approach to handling hash table misses
quickly, but that begs the question of why your design has so many hash
table misses that it is important to handle them quickly. In the Research
OS that I am working on (targetting PowerPC architecture, among others),
we assume that hash table misses are so infrequent, that we handle them as
in-core page faults. With a hash table 4 times the size of physical
memory, and a good spread of entries across them, this seems reasonable. I
got the impression that misses in your system are more frequent because
you allocate new VSIDs rather than unmap multiple pages from the page
table. If so, I guess that you can't be exploiting the dirty bit in the
page/hash table entry, and hence get double misses on write faults.
"We also disagree with one of your main conclusions: that
processors should not handle TLB misses in HW. I think that software
handling of TLB misses is an idea whose time as come ... and gone :-)
Hardware made sense in the past when you wanted to look at a whole pile of
entiries at the same time with specialized HW. Then, for a while it was
more efficient to do things in SW and avoid the HW complexity. Now, with
speculative execution and super-scaler highly pipelined processors,
handling them in SW means that you suffer a huge performance penalty
because you introduce a barrier/bubble on every TLB miss. With HW you can
freeze the pipeline and handle the miss with much reduced cost."
You and Paul and Victor did some excellent work, but you need to
keep in mind what implicit assumptions about processor design determined
whether the VMM design was an overall win. We can have a discussion about
whether the hardware improvements which make the VMM design less
adventageous are themselves the right strategy, but many commercial
processors are following that path after careful study of all options.
Your VMM design was correct for a specific, narrow class of
processors. We do not agree with your premise that the criteria for a
good processor design is whether it can utilize your VMM design.
As I said before, one needs to consider the microarchitecture
design and implementation of new processors before one can make sweeping
statements about which VMM design is best. You can create a wonderful
engineer solution, but are you solving the problem or simply masking a
symptom?
Cheers, David
===============================================================================
David Edelsohn T.J. Watson Research Center
dje@watson.ibm.com P.O. Box 218
+1 914 945 4364 (TL 862) Yorktown Heights, NY 10598
URL: http://www.research.ibm.com/people/d/dje/
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Gabriel Paubert <hidden> Date: 2001-02-08 23:28:41
On Thu, 8 Feb 2001, David Edelsohn wrote:
"Your paper discussess an approach to handling hash table misses
quickly, but that begs the question of why your design has so many hash
table misses that it is important to handle them quickly. In the Research
OS that I am working on (targetting PowerPC architecture, among others),
we assume that hash table misses are so infrequent, that we handle them as
in-core page faults. With a hash table 4 times the size of physical
memory, and a good spread of entries across them, this seems reasonable. I
got the impression that misses in your system are more frequent because
you allocate new VSIDs rather than unmap multiple pages from the page
table. If so, I guess that you can't be exploiting the dirty bit in the
page/hash table entry, and hence get double misses on write faults.
"We also disagree with one of your main conclusions: that
processors should not handle TLB misses in HW. I think that software
handling of TLB misses is an idea whose time as come ... and gone :-)
Hardware made sense in the past when you wanted to look at a whole pile of
entiries at the same time with specialized HW. Then, for a while it was
more efficient to do things in SW and avoid the HW complexity. Now, with
speculative execution and super-scaler highly pipelined processors,
handling them in SW means that you suffer a huge performance penalty
because you introduce a barrier/bubble on every TLB miss. With HW you can
freeze the pipeline and handle the miss with much reduced cost."
You and Paul and Victor did some excellent work, but you need to
keep in mind what implicit assumptions about processor design determined
whether the VMM design was an overall win. We can have a discussion about
whether the hardware improvements which make the VMM design less
adventageous are themselves the right strategy, but many commercial
processors are following that path after careful study of all options.
Your VMM design was correct for a specific, narrow class of
processors. We do not agree with your premise that the criteria for a
good processor design is whether it can utilize your VMM design.
As I said before, one needs to consider the microarchitecture
design and implementation of new processors before one can make sweeping
statements about which VMM design is best. You can create a wonderful
engineer solution, but are you solving the problem or simply masking a
symptom?
I agree with you, but that's only a gut feeling. Did you also notice that
Linux/PPC only uses half the recommended hash table size unless I'm
grossly mistaken ?
My feeling is that the hash table should be rather over- than under-sized,
especially with the amount of sharing there is between all the
applications running on "modern" desktops (large shared libraries for
X/KDE/GNOME among other things).
Regards,
Gabriel.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Paul Mackerras <hidden> Date: 2001-02-09 09:58:59
Gabriel Paubert writes:
I agree with you, but that's only a gut feeling. Did you also notice that
Linux/PPC only uses half the recommended hash table size unless I'm
grossly mistaken ?
My feeling is that the hash table should be rather over- than under-sized,
especially with the amount of sharing there is between all the
applications running on "modern" desktops (large shared libraries for
X/KDE/GNOME among other things).
We use a smaller than recommended hash table size for a couple of reasons:
- The hash table occupancy rates measured by Cort were very small,
typically less than 10% IIRC.
- A bigger hash table takes longer to clear when you have to do a
flush_tlb_all(). Fortunately there are only a couple of places
where flush_tlb_all is called, and they can both easily be changed
to flush_tlb_range. However, it is still necessary to clear the
hash table on a MMU context overflow.
- The recommended sizes are based on the idea that you have to try
quite hard to keep all the active PTEs in the hash table. We don't,
we can quickly fault PTEs into the hash table on demand so it is
less important for us to try to keep all the active PTEs in the hash
table.
Paul.
--
Paul Mackerras, Open Source Research Fellow, Linuxcare, Inc.
+61 2 6262 8990 tel, +61 2 6262 8991 fax
paulus@linuxcare.com.au, http://www.linuxcare.com.au/
Linuxcare. Support for the revolution.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Gabriel Paubert <hidden> Date: 2001-02-09 10:57:18
On Fri, 9 Feb 2001, Paul Mackerras wrote:
Gabriel Paubert writes:
quoted
I agree with you, but that's only a gut feeling. Did you also notice that
Linux/PPC only uses half the recommended hash table size unless I'm
grossly mistaken ?
My feeling is that the hash table should be rather over- than under-sized,
especially with the amount of sharing there is between all the
applications running on "modern" desktops (large shared libraries for
X/KDE/GNOME among other things).
We use a smaller than recommended hash table size for a couple of reasons:
- The hash table occupancy rates measured by Cort were very small,
typically less than 10% IIRC.
Then it means that the mm system is completely screwed up, even more than
I thought. I have to study first how VSID are handled, but this smells
definitively wrong.
- A bigger hash table takes longer to clear when you have to do a
flush_tlb_all(). Fortunately there are only a couple of places
where flush_tlb_all is called, and they can both easily be changed
to flush_tlb_range. However, it is still necessary to clear the
hash table on a MMU context overflow.
Yes, I was always worried by the added latency when a hash table clear
comes in. But the question is why do we have to do it ? Actually the
question is whether flush_tlb_all is even necessary.
- The recommended sizes are based on the idea that you have to try
quite hard to keep all the active PTEs in the hash table. We don't,
we can quickly fault PTEs into the hash table on demand so it is
less important for us to try to keep all the active PTEs in the hash
table.
I believe that this is a big mistake, faulting a PTE in the hash table is
an exception and wil never be as fast as having the entry already in the
PTE. And on SMP, you acquire hash_table_lock, an unnecessary variable BTW
but let us leave it for a later discussion, which will be very contended
and ping pong like mad between processors.
In short in all the previous discussion you had with Dan, I stand with
him and against you in all and every single aspect, except for the TLB
preloading thing.
Regards,
Gabriel.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Paul Mackerras <hidden> Date: 2001-02-09 11:26:45
Gabriel Paubert writes:
quoted
- The hash table occupancy rates measured by Cort were very small,
typically less than 10% IIRC.
Then it means that the mm system is completely screwed up, even more than
I thought. I have to study first how VSID are handled, but this smells
definitively wrong.
Gabriel: one word: Measure. Then criticize, if you like.
Cort's measurements were done with lmbench and kernel compiles IIRC.
Don't forget that not all pages of physical memory are used via HPTEs;
many pages are used for page-cache pages of files which are read and
written rather than being mmap'd. For example, in the case of a
kernel compile you would hopefully have all of the relevant kernel
source and object files in the page cache but never mmap'd, and those
pages would be typically be accessed through the kernel BAT mapping.
And the other point is that the recommended hash table sizes are large
enough to map the whole of physical memory 4 times over.
Yes, I was always worried by the added latency when a hash table clear
comes in. But the question is why do we have to do it ? Actually the
question is whether flush_tlb_all is even necessary.
As I have already said, flush_tlb_all can be avoided completely, but
the flush on MMU context overflow is necessary (but fortunately very
rare).
I believe that this is a big mistake, faulting a PTE in the hash table is
an exception and wil never be as fast as having the entry already in the
PTE. And on SMP, you acquire hash_table_lock, an unnecessary variable BTW
but let us leave it for a later discussion, which will be very contended
and ping pong like mad between processors.
Measurements?
(David, perhaps you could comment on the need for hash_table_lock and
the possible hardware deadlocks if you do tlbie/tlbsync on different
CPUs at the same time?)
Like everything, the hash table size is a tradeoff. A big hashtable
has disadvantages, one of which is that less of it will fit in L2
cache and thus TLB misses will take longer on average.
In short in all the previous discussion you had with Dan, I stand with
him and against you in all and every single aspect, except for the TLB
preloading thing.
I look forward to seeing your benchmark results for the different
alternatives.
Paul.
--
Paul Mackerras, Open Source Research Fellow, Linuxcare, Inc.
+61 2 6262 8990 tel, +61 2 6262 8991 fax
paulus@linuxcare.com.au, http://www.linuxcare.com.au/
Linuxcare. Support for the revolution.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/