kernel mapping

How come we don't use iopa() and friends for all kernel mapping
information?  It is only defined for CONFIG_APUS, but is the right
thing to use on 8xx and 4xx, and probably all processors.  The
virt_to_bus/bus_to_virt contain the quickie arithmetic hack with
KERNELBASE, but that isn't the right thing to do for any kmalloc()
or valloc() space or if you don't have BAT mapping.

I am considering making these functions more generic, removing the
#ifdefs, and implementing "simulated" BAT mapping for processors
like the 8xx and 4xx that don't have BATs (not for 2.4, of course :-).

Why shouldn't I do this?

	-- Dan

--

	I like MMUs because I don't have a real life.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Dan Malek wrote:

How come we don't use iopa() and friends for all kernel mapping
information?  It is only defined for CONFIG_APUS, but is the right
thing to use on 8xx and 4xx, and probably all processors.  The
virt_to_bus/bus_to_virt contain the quickie arithmetic hack with
KERNELBASE, but that isn't the right thing to do for any kmalloc()
or valloc() space or if you don't have BAT mapping.

I am considering making these functions more generic, removing the
#ifdefs, and implementing "simulated" BAT mapping for processors
like the 8xx and 4xx that don't have BATs (not for 2.4, of course :-).

Why shouldn't I do this?

        -- Dan

For the 405 I had to use iopa() for virt_to_bus() because there are
cases where I create a virtual address for IO buffers that is
uncached, and that virtual address is not (physical address + KERNELBASE).
I also have the beginnings of simulated BAT mapping for the 405
(not quite there, but part way).

-Frank
--
Frank Rowand [off-list ref]
MontaVista Software, Inc

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Frank Rowand wrote:

I also have the beginnings of simulated BAT mapping for the 405
(not quite there, but part way).

I know.  I'm currently re-writing it all to be generic.  All of
the 4xx specific pinned entry stuff is gone.  I've been thinking
about this too long for the 8xx and now have a reason to do it.
Some of the logic is in the iopa and mm_ptov functions now, which
will also just work fine on the 6xx/7xx/7xxx.


	-- Dan

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Why do we need simulated bat registers.

Chip

Frank Rowand wrote:

Dan Malek wrote:

quoted

How come we don't use iopa() and friends for all kernel mapping
information?  It is only defined for CONFIG_APUS, but is the right
thing to use on 8xx and 4xx, and probably all processors.  The
virt_to_bus/bus_to_virt contain the quickie arithmetic hack with
KERNELBASE, but that isn't the right thing to do for any kmalloc()
or valloc() space or if you don't have BAT mapping.

I am considering making these functions more generic, removing the
#ifdefs, and implementing "simulated" BAT mapping for processors
like the 8xx and 4xx that don't have BATs (not for 2.4, of course :-).

Why shouldn't I do this?

        -- Dan

For the 405 I had to use iopa() for virt_to_bus() because there are
cases where I create a virtual address for IO buffers that is
uncached, and that virtual address is not (physical address + KERNELBASE).
I also have the beginnings of simulated BAT mapping for the 405
(not quite there, but part way).

-Frank
--
Frank Rowand [off-list ref]
MontaVista Software, Inc

Ralph Blach wrote:

Why do we need simulated bat registers.

To improve performance.  Right now, on the 4xx there is the
concept of "pinned" TLB entries to reduce/eliminate TLB misses
on large mapped areas (like kernel text/data or I/O).  The 8xx
does this in some custom applications as well.  These are just
hacks that are headed down a disastrous maintenance path that
need to be stopped now for a more generic solution.

I have been experimenting with many different methods of using
the "large" page table sizes through the generic memory management
methods that already exist in the kernel.  I believe I can wrap
the concept of the pinned TLB entries into the same logic as BAT
register management on the bigger processors.  Hence, I call them
simulated BAT registers....the semantics aren't quite the same.

The BAT registers are a really good thing, and although the large
page size TLB entries are more flexible, they require more software
overhead.  I would like to make some generic Linux MM modifications
to help us support variable page sizes, but I suspect that will
never happen.


	-- Dan

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Dan,

Thanks for the info.  I agree that Pinned tlbs could be maintence
headache with each 4xx/8xx
chip requiring a different set of pinned tlbs.

Chip

Dan Malek wrote:

Ralph Blach wrote:

quoted

Why do we need simulated bat registers.

To improve performance.  Right now, on the 4xx there is the
concept of "pinned" TLB entries to reduce/eliminate TLB misses
on large mapped areas (like kernel text/data or I/O).  The 8xx
does this in some custom applications as well.  These are just
hacks that are headed down a disastrous maintenance path that
need to be stopped now for a more generic solution.

I have been experimenting with many different methods of using
the "large" page table sizes through the generic memory management
methods that already exist in the kernel.  I believe I can wrap
the concept of the pinned TLB entries into the same logic as BAT
register management on the bigger processors.  Hence, I call them
simulated BAT registers....the semantics aren't quite the same.

The BAT registers are a really good thing, and although the large
page size TLB entries are more flexible, they require more software
overhead.  I would like to make some generic Linux MM modifications
to help us support variable page sizes, but I suspect that will
never happen.

        -- Dan

Dan Malek wrote:

Ralph Blach wrote:

quoted

Why do we need simulated bat registers.

To improve performance.  Right now, on the 4xx there is the
concept of "pinned" TLB entries to reduce/eliminate TLB misses
on large mapped areas (like kernel text/data or I/O).  The 8xx
does this in some custom applications as well.  These are just
hacks that are headed down a disastrous maintenance path that
need to be stopped now for a more generic solution.

At the moment, the 405 processors _require_ kernel memory to be
pinned because the tlb miss handlers use virtual addresses.  When
I started the 405 port I planned to move the TLB handlers into
assembly running in real mode.  Then when I started seeing info
about the 440 I backed away from that plan because the 440 always
runs with the MMU enabled.  I'm still thinking about the 440...

I pinned some IO ranges as a convenience when I was first porting
to the 405gp but plan to remove those pins.  Though I'm somewhat
tempted to leave a pin in place for the on-chip ethernet device
if performance measurements show a significant gain.

I have been experimenting with many different methods of using
the "large" page table sizes through the generic memory management
methods that already exist in the kernel.  I believe I can wrap
the concept of the pinned TLB entries into the same logic as BAT
register management on the bigger processors.  Hence, I call them
simulated BAT registers....the semantics aren't quite the same.

I think that's a good idea.  If you do so, please provide a way to
force an entry to be locked in the tlb.

The BAT registers are a really good thing, and although the large
page size TLB entries are more flexible, they require more software
overhead.  I would like to make some generic Linux MM modifications
to help us support variable page sizes, but I suspect that will
never happen.

        -- Dan

I've toyed with the variable pages sizes idea too, and it just hasn't
moved up high enough on my priority list.  I'm not sure I'm quite as
pessimistic as you about whether it will ever happen because several
other architecture support variable page sizes (including pa-risc
and (I think) IA-64).

-Frank
--
Frank Rowand [off-list ref]
MontaVista Software, Inc

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Frank Rowand wrote:

At the moment, the 405 processors _require_ kernel memory to be
pinned because the tlb miss handlers use virtual addresses.

I changed that, too.  It works like the other processors, in
particular, the 8xx.

.........  Then when I started seeing info
about the 440 I backed away from that plan because the 440 always
runs with the MMU enabled.  I'm still thinking about the 440...

No way....Dammit can't you IBM guys follow your own rules :-).

I pinned some IO ranges as a convenience when I was first porting
to the 405gp but plan to remove those pins.

Those are actually performace advantages, and I am doing that
on some 8xx applications.  The difference now is we don't have
to actually allocate specific "pinned" entries, the large mapping
will just happen as part of the TLB reload.

I think that's a good idea.  If you do so, please provide a way to
force an entry to be locked in the tlb.

Nope.  I don't want to do that.  Then you have to make processor
specific trade offs, or incur high management overhead like the
405 does now.  For example, some of the processors allow a fixed
number of locked entries, but you have to trade off what you will
put there against losing TLB entries.  Or, you do like the 405
does and create a "software" locking, losing the use of some
very functional TLB management instructions.

By not locking entries and using large page table entries you don't
need to have processor unique configurations that are cumbersome
or unworkable on lesser featured processors.  You also let the
system operation find the best distribution of TLB entries.  Yes,
there is a clearly visible latency concern with loading TLBs, but
considering the amount of context we are switching these days a
single large page TLB miss is insignificant.

I've toyed with the variable pages sizes idea too, and it just hasn't
moved up high enough on my priority list.  I'm not sure I'm quite as
pessimistic as you about whether it will ever happen

It's going to happen with the 405 merge.  It has to because I have
already screwed up and coded myself into a corner, and I want the
same features on the 8xx already as well.


	-- Dan

--

	I like MMUs because I don't have a real life.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Dan Malek wrote:

Frank Rowand wrote:

I have only a small amount of performance instrumentation and measurements.
Some of what I have to say is based on observation, inference, and
conjecture...

quoted

I pinned some IO ranges as a convenience when I was first porting
to the 405gp but plan to remove those pins.

Those are actually performace advantages, and I am doing that
on some 8xx applications.  The difference now is we don't have
to actually allocate specific "pinned" entries, the large mapping
will just happen as part of the TLB reload.

The IO ranges that I pinned were all just a 4k page (except the 64k
"page" for PCI IO space, which shouldn't be accessed much except for
PCI device initialization).  So the only performance advantage I
gained was avoiding TLB misses, not from large pages.

quoted

I think that's a good idea.  If you do so, please provide a way to
force an entry to be locked in the tlb.

Nope.  I don't want to do that.  Then you have to make processor
specific trade offs, or incur high management overhead like the
405 does now.  For example, some of the processors allow a fixed
number of locked entries, but you have to trade off what you will
put there against losing TLB entries.  Or, you do like the 405
does and create a "software" locking, losing the use of some
very functional TLB management instructions.

The 405 core (and thus the many processors based on it) has a 64 entry
tlb.  While debugging via a JTAG debugger I have observed that the
tlb very quickly gets filled with entries for the current context.  It
is extremely rare to see entries for a different context left over.

From this, I infer that the tlb is not large enough to hold a working

set.  (If I was still working as a performance geek, I would find this
an interesting area to instrument.)  Locking a few kernel entries in
the tlb means that the majority of the kernel's working set _is_ in
the tlb at all times.  Here is a simple measurement of tlb misses
(running a simple load of copying nfs mounted files around, etc):

  dtlb  misses:   34679326        <--- data tlb
  itlb  misses:   33075725        <--- instruction tlb
  d + i misses:   67755051
  ktlb  misses:     233683        <--- kernel addresses
  utlb  misses:   67521368        <--- user space addresses
  k + u misses:   67755051


If you want to repeat the measurement with other workloads, just
cat /proc/ppc_htab in my kernel to get the above data.

For the 405, the only tlb management instruction I sacrificed was
the tlbia (invalidate the entire tlb) that I would have used for
PPC4xx_tlb_flush_all(), which is used by flush_tlb_all(), which
is only called from:

  ppc_htab_write()
  mmu_context_overflow()
  vmfree_area_pages()
  vmalloc_area_pages()
  flush_all_zero_pkmaps()

Which doesn't seem to be much of a sacrifice for a large gain.

By not locking entries and using large page table entries you don't
need to have processor unique configurations that are cumbersome
or unworkable on lesser featured processors.  You also let the
system operation find the best distribution of TLB entries.  Yes,

The tlb is not large enough to accumulate a working set in the tlb,
so system operation never finds the best distribution of tlb entries.

there is a clearly visible latency concern with loading TLBs, but
considering the amount of context we are switching these days a
single large page TLB miss is insignificant.

It will be nice to have large page TLB implemented.

-Frank
--
Frank Rowand [off-list ref]
MontaVista Software, Inc

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Frank Rowand wrote:

The 405 core (and thus the many processors based on it) has a 64 entry
tlb.  While debugging via a JTAG debugger I have observed that the
tlb very quickly gets filled with entries for the current context.

Well, that's because we force it to do that.  Programs that have
a huge working set and run for an extended period will fill the
TLB.  Programs with small working sets will not if we actually
use the contexts properly, but we don't.  The way contexts are
used today, it is effectively flushing the TLB on every switch.
We do the same thing with VSIDs on the "bigger" processors, and
this isn't right either.  The Linux VM properly manages memory
contexts, and we should extend this into the PowerPC specific
software.

I have an LRU context algorithm for the 8xx, and I want to extend
this into the 4xx and even the other processors.  The idea is right,
but I need something that will scale beyond the small 16 contexts
of the 8xx.  I just don't have that yet.


	-- Dan

--

	I like MMUs because I don't have a real life.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/