8xx MMU Table Walk Base (was Re: kernel crashes at InstructionTLBMiss )

       mfspr   r20, M_TWB      /* Get level 1 table entry address */

...

As you can see, r20 is 400f1c00, which looks wrong, but why? Any suggestions?

At this point, the MMU is disabled so r20, which is loaded from the MMU Table
Walk Base register, should be a physical address - 400f1c00 is not a likely
physical address for something in RAM (unless you have a weird disjoint RAM
setup), so yes it certainly looks wrong. Incorrect TWB contents is a disaster.

Here we come to a dilemma that I have had since I started with this stuff.
I have never been able to get an 8xx kernel running without adding a patch
to update the Table Walk Base register at the time that a new mm context is
activated.

Let me explain: normally the TWB is loaded at context switch time which makes
sense because a different task with a different virtual memory context will be
running. This is done in the following code in the _switch function in
arch/ppc/kernel/entry.S:

tophys(r0,r4)
mtspr	SPRG3,r0	/* Update current THREAD phys addr */
#ifdef CONFIG_8xx
/* XXX it would be nice to find a SPRGx for this on 6xx,7xx too */
lwz	r9,PGDIR(r4)	/* cache the page table root */
       tophys(r9,r9)		/* convert to phys addr */
       mtspr   M_TWB,r9	/* Update MMU base address */
tlbia
SYNC
#endif /* CONFIG_8xx */

The contents of the TWB should be the address stored in current->thread.pgdir
converted to a physical address. The above code is the only place that the
TWB is written to, anywhere in the kernel (that I can find). The TWB is then
used in the TLB miss handlers to load the TLB entry (assuming a mapping exists
- if not, do_page_fault() is called to fill it in).

But I have found that there is a situation during "exec()" where a newly
created mm context is "activated" (via activate_mm() in asm/mmu_context.h)
before the task is actually "switch"ed to (presumably to copy the arguments
and environment etc from the old task - which is being overwritten) i.e. the
TWB is not updated because a switch hasn't occurred [NOTE: this is only my
theory - I am not an expert on this stuff]

Without my patch, the exec of "/sbin/init" hangs in an endless TLBMiss handler
loop, where a virtual address is accessed which causes a TLB miss, the TWB
has contents of the old pgdir which does not have a mapping for that virtual
address so do_page_fault() is called to fill it in, but do_page_fault()
decides that that mapping exists and everything is ok so why the hell did
you call me, I'll just return doing nothing! - the access is re-tried which
causes a TLB miss again at the same virtual address. The kernel is in a
dead hang (although later 2.[34].* kernels exhibit different symptoms, which
mystifies me a bit - i.e. characters typed on the console are echoed, and I
know timer interrupts are occuring, because I have a rotating thingy on the
LCD display which updates once a second via the timer interrupt handler, so it
is not a complete dead hang).

The patch I always have to add to arch/ppc/kernel/head_8xx.S is:

  */
 _GLOBAL(set_context)
         mtspr   M_CASID,r3		/* Update context */
+	lwz	r3, THREAD+PGDIR(r2)
+	tophys(r3, r3)
+	mtspr	M_TWB, r3
         tlbia
 	SYNC
 	blr

I know this is wrong, but it seems to work for me (unless the TWB can be
considered to be part of the MMU context, and therefore it is legitimate
to update it in set_context()? I don't know).

I have tried other things e.g. adding a "set_context_and_twb()" function,
just after the set_context() function (without above patch), e.g.:

--- arch/ppc/kernel/head_8xx.S	2000/04/28 06:35:05	1.1.1.5
+++ arch/ppc/kernel/head_8xx.S	2000/06/05 07:51:50

@@ -905,6 +905,19 @@
 	SYNC
 	blr

+/*
+ * the 8xx tablewalk base register (M_TWB) must be consistent with
+ * the currently active mm. This is called from switch_mm() and
+ * activate_mm() in include/asm-ppc/mmu_context.h
+ */
+_GLOBAL(set_context_and_twb)
+        mtspr   M_CASID,r3		/* Update context */
+	tophys(r4, r4)
+	mtspr	M_TWB, r4
+	tlbia
+	SYNC
+	blr
+
 /* Jump into the system reset for the rom.
  * We first disable the MMU, and then jump to the ROM reset address.
  *

Then doing something like this:

--- include/asm-ppc/mmu_context.h	2000/03/07 03:59:54	1.1.1.2
+++ include/asm-ppc/mmu_context.h	2000/06/05 07:46:35

@@ -52,6 +52,11 @@
 extern void set_context(int context);

 #ifdef CONFIG_8xx
+/* same as above plus loads the 8xx tablewalk base register also */
+extern void set_context_and_twb(int, void *);
+#endif
+
+#ifdef CONFIG_8xx
 extern inline void mmu_context_overflow(void)
 {
 	atomic_set(&next_mmu_context, -1);

@@ -85,7 +90,10 @@
 {
 	tsk->thread.pgdir = next->pgd;
 	get_mmu_context(next);
-	set_context(next->context);
+	if (tsk == current)
+		set_context_and_twb(next->context, tsk->thread.pgdir);
+	else
+		set_context(next->context);
 }

 /*

@@ -96,7 +104,7 @@
 {
 	current->thread.pgdir = mm->pgd;
 	get_mmu_context(mm);
-	set_context(mm->context);
+	set_context_and_twb(mm->context, current->thread.pgdir);
 }

 /*

This works also, though I'm not sure about it. I was thinking that maybe the
set_context() in switch_mm() should only be done if the switch_mm() is being
performed on the "current" task. e.g.

	tsk->thread.pgdir = next->pgd;
	get_mmu_context(next);
	if (tsk == current)
		set_context(next->context);

Then set_context() could simply update the TWB with current->thread.pgdir.
But I think the only place switch_mm() is called is in the task context
switch code anyway, which means current is about to change, and also means
I get confused :-) But I know activate_mm() is used in other places -
something to do with "lazy tlb" mode, and also in exec(). I give up.

One thing I think is for certain in all this - do_page_fault() should
*NEVER* return without having done something - anything - to ensure
that the same fault does not re-occur after the handler returns - if it
can't handle the fault, it should either kill the task if it is in user
mode, or panic if in kernel mode.

One thing that bothers me is why this behaviour only occurs for me? I have
no idea, but obviously it is only me, otherwise no-one would have a working
embedded 8xx 2.[34].* kernel. I suspect I am doing something else which
triggers this bug, or else there is something I don't understand
(not unlikely :-).

Note: I have only ever tried the 2.[34].* series of kernels. I have not
tried the 2.2.* kernels, but some code snippets I have seen in the list
archives suggest ... I just searched the list and found the following
comment from Dan Malek on 16 Dec 98:

	> BTW, why must the M_TWB be set in SET_PAGE_DIR ?

	The M_TWB points to the first level page table (Linux pgd_t)
	and is used in the mpc8xx page fault handler.  When Linux
	deletes or otherwise modifies the memory map object such that
	the first level page table is modified (as during exec), it
	uses SET_PAGE_DIR.  Since the first level table has potentially
	moved to a new memory location, we have to set M_TWB at
	this time.  If we don't, a process exec without an intervening
	context switch will cause us to use a bogus M_TWB when
	trying to find page tables.


	    -- Dan

OK - where is SET_PAGE_DIR() in the 2.[34].* kernels? Following the threads
it appears that this discussion was had a long time ago, but in the other
direction - the TWB was being updated too often, and the consensus was that
it should only be updated when the SET_PAGE_DIR macro was setting the page
dir for the current task. Now it is not setting it at all.

I think I'd better shut up now and let other more experienced people tell me
what I have missed or where I have gone wrong :-) Cheers!
								Murray...
--
Murray Jensen, CSIRO Manufacturing Sci & Tech,         Phone: +61 3 9662 7763
Locked Bag No. 9, Preston, Vic, 3072, Australia.         Fax: +61 3 9662 7853
Internet: Murray.Jensen@cmst.csiro.au  (old address was mjj@mlb.dmt.csiro.au)

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Murray Jensen wrote:

Here we come to a dilemma that I have had since I started with this stuff.
I have never been able to get an 8xx kernel running without adding a patch
to update the Table Walk Base register at the time that a new mm context is
activated.

After reading your diatribe perhaps I should provide a little information.
There are many subtle changes to context switching that happen during
the minor updates (which could be weekly).  There are several patches
floating around (and probably more kernel sources) that certainly
are not correct.  I don't know where you get your source code, but there
are exactly two consistent and working kernel sources that I have ever
provided.  One is in ftp://linuxppc.cs.nmt.edu/pub/linuxppc/embedded,
the mpc8xx-2.2.13.tgz tarball.  A better and completely up to date
kernel is in ftp.mvista.com/pub/CDK/wip/ppc_8xx/RPMS (along with
everything else to build an 8xx embedded system).  Everyone should be
using the kernel from MontaVista, and if something isn't in there
that you want, send me patches against that.

There are patches posted against that original tarball, and make sure
you are not mixing kernel versions and patches.

Finally, lots of bugs associated with porting to new hardware manifest
themselves as "problems" in any VM related function.  Since many people
don't understand the subtle interactions of all of these functions (as
evidenced by your message) you become convinced the problem is associated
with this complexity and fail to unravel the clues to the real cause.
This could be as simple as intrusive debugging hardware, some silicon
bug not understood, or prototype hardware not working correctly.

There are lots of products and systems in development running this software,
so you have to approach this generic software from the assumption that
it is first likely to be working.  You seldom hear from those people.
Are there possible bugs?  Sure, and you have to provide minimal information
for the rest of us to help out.  Where did you get the sources? What
patches did you apply?  What are your hardware details?  What
modifications did you make?


As for 2.4.xx, the 8xx still doesn't work correctly.  However, I
discovered it failed to work after the 403 additions, so I am now
learning about the 403 in an effort to make everything live happily
together again.  Note, this has nothing to do with M_TWB......


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Does anybody know how to write a net driver for a 2.2 kernel?

Rubini's book _Linux_Device_Drivers_ only covers the 2.1 kernel,
and significant changes have been made since then.

I have essentially identical drivers on board and host, but
they behave differently -- the PowerPC version works but the i386
does not.

In particular, I have looked through the kernel code and made the
driver no longer crash the kernel. But I cannot get the host's
network stack to accept packets received by my driver; the powerpc
Linux stack accepts them just fine.

My configuration is that I boot a PowerPC board (with Linux
2.2.15-2.9.0) from an i386 host running RedHat Linux 2.2.14-5.0.
I have a SIO driver on both host and board which becomes the console
of the board's Linux, and on the host the boot program becomes a
cheap terminal emulator so I can issue commands to the board's Linux
and see its printk's and responses. The on-board linux comes up
fine and my init script configures and starts the network device.

ifconfig of my device on the host shows packets received, but
netstat -s shows IP did not get them. When I dump the skb the data
it contains looks at least superficially like a valid IP datagram
(the 3rd word is the IP address of the board, and the 4th word is
the IP address of the host [count words from 0]).

The weird thing is that when I ping my PowerPC board from the
i386 host, the packets are received on the PowerPC Linux just fine,
and they are returned just fine, but the host does not see them
after the driver calls netif_rx(). On the PowerPC "netstat -s"
shows all packets received and sent by both IP and ICMP; on the
host neither IP nor ICMP sees any packets. On both PowerPC and
host, "ifconfig lspsnet" shows the right number of packets sent
and received. And debugging printk-s of the skb just before the
call to netif_rx() are quite similar on the PowerPC and host --
the IP addresses are interchanged as expected.


What I think are the relevant details: this is not an ethernet;
my do_rcvpkt() is called every tick using a timer; it checks for
a packet arrival (in a memory buffer), and when one arrives:
	// data = pointer to the packet data
	// len = length of the packet data (# bytes)
	/* allocate and fill a skb */
        len4 = (len+3) & ~3;
        skb = dev_alloc_skb(len4);
        if(!skb) return (npkt ? 0 : -ENOMEM);
	memcpy(skb_put(skb,len),data,len);
        skb->dev = dev;
        skb->protocol = ETH_P_IP;
        skb->pkt_type = PACKET_HOST;
        skb->ip_summed = CHECKSUM_UNNECESSARY;
        /* all packets received are for us -- fake mac addr */
        skb->mac.raw = skb_push(skb,dev->addr_len);
        memcpy(skb->mac.raw,dev->dev_addr,dev->addr_len);
        skb_pull(skb,dev->addr_len);
	// deliver skb to higher layers
        netif_rx(skb);
	// update counter and statistics
	++npkt;
	dev->last_rx = jiffies;
        ++Enet_stats.rx_packets;
        Enet_stats.rx_bytes += len;

Note the attempt to fake out the mac address (mac.raw must point
between skb->head and skb->data). My dev->addr_len is 1, but
changing it to 6 did not help. Printing the skb contents shows that
the drivers do indeed transfer the data unchanged, and I think
I set all the required skb fields above.... WHAT AM I MISSING???


Tom Roberts	tjroberts@lucent.com

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/