fix swapping on 8xx?

Hi, 

The following is an attempt to fix swapping on 8xx by not touching
_PAGE_ACCESSED bit if the page is not present.

Dan, what do you think?

diff --git a/arch/ppc/kernel/head_8xx.S b/arch/ppc/kernel/head_8xx.S
index de09787..4451828 100644
--- a/arch/ppc/kernel/head_8xx.S
+++ b/arch/ppc/kernel/head_8xx.S

@@ -41,6 +41,22 @@
 #else
 #define DO_8xx_CPU6(val, reg)
 #endif
+
+/* can't overwrite the _PAGE_ACCESSED bit of a non-present page (might
+   contain swap data). */
+#ifdef CONFIG_SWAP
+#define SET_PAGE_ACCESSED(reg, destreg, tmpreg)		\
+	andi.	tmpreg, reg, _PAGE_PRESENT;		\
+	beq	9f;					\
+	ori	reg, reg, _PAGE_ACCESSED;		\
+	stw	r10, 0(destreg);			\
+9:
+#else
+#define SET_PAGE_ACCESSED(reg, destreg, tmpreg)	\
+	ori	reg, reg, _PAGE_ACCESSED;	\
+	stw	reg, 0(destreg);
+#endif
+
 	.text
 	.globl	_stext
 _stext:

@@ -332,8 +348,7 @@ InstructionTLBMiss:
 	mfspr	r11, SPRN_MD_TWC	/* ....and get the pte address */
 	lwz	r10, 0(r11)	/* Get the pte */
 
-	ori	r10, r10, _PAGE_ACCESSED
-	stw	r10, 0(r11)
+	SET_PAGE_ACCESSED(r10, r11, r3)
 
 	/* The Linux PTE won't go exactly into the MMU TLB.
 	 * Software indicator bits 21, 22 and 28 must be clear.

@@ -399,8 +414,7 @@ DataStoreTLBMiss:
 	mtspr	SPRN_MD_TWC, r11
 
 	mfspr	r11, SPRN_MD_TWC	/* get the pte address again */
-	ori	r10, r10, _PAGE_ACCESSED
-	stw	r10, 0(r11)
+	SET_PAGE_ACCESSED(r10, r11, r3)	/* and update pte in table */
 
 	/* The Linux PTE won't go exactly into the MMU TLB.
 	 * Software indicator bits 21, 22 and 28 must be clear.

@@ -507,9 +521,9 @@ DataTLBError:
 
 	/* Update 'changed', among others.
 	*/
-	ori	r10, r10, _PAGE_DIRTY|_PAGE_ACCESSED|_PAGE_HWWRITE
+	ori	r10, r10, _PAGE_DIRTY|_PAGE_HWWRITE
 	mfspr	r11, SPRN_MD_TWC		/* Get pte address again */
-	stw	r10, 0(r11)		/* and update pte in table */
+	SET_PAGE_ACCESSED(r10, r11, r3)		/* and update pte in table */
 
 	/* The Linux PTE won't go exactly into the MMU TLB.
 	 * Software indicator bits 21, 22 and 28 must be clear.

On Nov 7, 2005, at 10:10 AM, Marcelo Tosatti wrote:

The following is an attempt to fix swapping on 8xx by not touching
_PAGE_ACCESSED bit if the page is not present.

Ugh ....  I suppose.  I hate assembler code macros .......
Somehow, "swapping" and "8xx" just don't belong together.
I'm tempted to add a configuration option that is the complete
opposite of this and assumes are really embedded system.
Mark pages as always accessed, data pages as always dirty,
and you can eliminate lots of TLB faults in systems that are
fairly static.

None of this should really be done in a TLB miss handler,
if there is anything that needs to be updated in a PTE, it should
be pushed out to an error function.  A TLB miss handler
should fit in a cache line ......

	-- Dan

On Tuesday 08 November 2005 01:55, Dan Malek wrote:

On Nov 7, 2005, at 10:10 AM, Marcelo Tosatti wrote:

quoted

The following is an attempt to fix swapping on 8xx by not touching
_PAGE_ACCESSED bit if the page is not present.

Ugh ....  I suppose.  I hate assembler code macros .......
Somehow, "swapping" and "8xx" just don't belong together.

Well, at least it sounds ugly together, but it is also at least conceiveable. 
There seem to be people who use PCMCIA for an IDE interface, so swapping may 
become desireable in some cases.

I'm tempted to add a configuration option that is the complete
opposite of this and assumes are really embedded system.
Mark pages as always accessed, data pages as always dirty,
and you can eliminate lots of TLB faults in systems that are
fairly static.

It sounds tempting indeed, but should you really notice a performance increase 
out of this?

Greetings,

-- 
David Jander

On Tue, Nov 08, 2005 at 01:59:26PM +0200, David Jander wrote:

On Tuesday 08 November 2005 01:55, Dan Malek wrote:

quoted

On Nov 7, 2005, at 10:10 AM, Marcelo Tosatti wrote:

quoted

The following is an attempt to fix swapping on 8xx by not touching
_PAGE_ACCESSED bit if the page is not present.

Ugh ....  I suppose.  I hate assembler code macros .......
Somehow, "swapping" and "8xx" just don't belong together.

Well, at least it sounds ugly together, but it is also at least conceiveable. 
There seem to be people who use PCMCIA for an IDE interface, so swapping may 
become desireable in some cases.

I think Dan might be in the camp that says a properly designed embedded
system won't need to swap.  And when I hear about how people do try and
swap on systems like this, I really start agreeing.  Maybe we could make
8xx just select SWAP=n? :)

quoted

I'm tempted to add a configuration option that is the complete
opposite of this and assumes are really embedded system.
Mark pages as always accessed, data pages as always dirty,
and you can eliminate lots of TLB faults in systems that are
fairly static.

It sounds tempting indeed, but should you really notice a performance increase 
out of this?

Compared to 8xx in 2.6 today?  Absolutely.

-- 
Tom Rini
http://gate.crashing.org/~trini/

On Nov 8, 2005, at 12:56 PM, Tom Rini wrote:

I think Dan might be in the camp that says a properly designed embedded
system won't need to swap.

I'm actually in the camp that knows the majority of systems
running Linux aren't workstations and don't have disk drives.
It would be nice to have selectable features for such applications.

quoted

It sounds tempting indeed, but should you really notice a performance increase
out of this?

Yes.  Most importantly it adds some predictability which is quite
measurable on this class of processor.  It was a coding mistake way
back in 2.2 that actually exposed this.  I was taking some liberties by
updating more status in the PTE than I should have, and Paulus
made me fix it in 2.4 :-)  It basically eliminates TLB faults associated
with system status tracking, since we aren't paging there isn't any
need to track page aging and such.  If you are paging, it still
functions properly, you may just not pick the most suitable pages
to steal.

Thanks.

	-- Dan

quoted

I think Dan might be in the camp that says a properly designed embedded
system won't need to swap.

I'm actually in the camp that knows the majority of systems
running Linux aren't workstations and don't have disk drives.
It would be nice to have selectable features for such applications.

Well, me too, but there are people out there trying to do strange stuff. I
remember myself even once trying to get swapping to work over nfs, because
I wanted to compile perl5 natively (overnight of course), since
cross-compiling perl is quite a bit of pain. Perl intends to
"include-together" enormous source files, that exhausted the whole 64Mbyte
RAM on my embedded board ;-)
Needless to say, swap over nfs did never work reliably enough to even
start compiling.

quoted

quoted

It sounds tempting indeed, but should you really notice a performance
increase out of this?

Yes.  Most importantly it adds some predictability which is quite
measurable on this class of processor. It was a coding mistake way back
in 2.2 that actually exposed this.  I was taking some liberties by
updating more status in the PTE than I should have, and Paulus
made me fix it in 2.4 :-)  It basically eliminates TLB faults associated
with system status tracking, since we aren't paging there isn't any need
to track page aging and such.  If you are paging, it still
functions properly, you may just not pick the most suitable pages to
steal.

FYI, I have some benchmark results of yesterday's git of kernel-2.6 (with
_tlbie(address); just moved out of the two innermost if()'s in the
function update_mmu_cache()) compared to kernel 2.4.25 from denx, made
with "nbench". The results are quite interesting, and contradicting
(older) benchmark results, Wolfgang made a few months back:

kernel-2.6.14-git-20051107:
---------------------------
TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          32.654  :       0.84  :       0.28
STRING SORT         :          1.7408  :       0.78  :       0.12
BITFIELD            :      8.3466e+06  :       1.43  :       0.30
FP EMULATION        :           3.506  :       1.68  :       0.39
IDEA                :           115.3  :       1.76  :       0.52
HUFFMAN             :          27.855  :       0.77  :       0.25
LU DECOMPOSITION    :         0.35932  :       0.02  :       0.01

kernel-2.4.25-devel-denx:
-------------------------
TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          30.438  :       0.78  :       0.26
STRING SORT         :          1.5842  :       0.71  :       0.11
BITFIELD            :      7.9506e+06  :       1.36  :       0.28
FP EMULATION        :           3.258  :       1.56  :       0.36
IDEA                :          108.89  :       1.67  :       0.49
HUFFMAN             :          26.281  :       0.73  :       0.23
LU DECOMPOSITION    :         0.32765  :       0.02  :       0.01

This is all on the same hardware: MPC852T + 32Mbyte SDRAM running from
NFS. kernel 2.6 is faster all the way through!!
The results so far are pretty encouraging I'd say. Maybe something like
lmbench would help to shed more light on details important for kernel
testing, like context-switch overhead, mm, etc...
Nbench is quite useless here (a single task doing all sorts of nonsense,
almost no critical syscalls), but that's what makes the enormous
differences even more remarkable IMO. Maybe something is broken in 2.4?

Greetings,

-- 
David Jander

On Tue, Nov 08, 2005 at 10:56:58AM -0700, Tom Rini wrote:

On Tue, Nov 08, 2005 at 01:59:26PM +0200, David Jander wrote:

quoted

On Tuesday 08 November 2005 01:55, Dan Malek wrote:

quoted

On Nov 7, 2005, at 10:10 AM, Marcelo Tosatti wrote:

quoted

The following is an attempt to fix swapping on 8xx by not touching
_PAGE_ACCESSED bit if the page is not present.

Ugh ....  I suppose.  I hate assembler code macros .......
Somehow, "swapping" and "8xx" just don't belong together.

Well, at least it sounds ugly together, but it is also at least conceiveable. 
There seem to be people who use PCMCIA for an IDE interface, so swapping may 
become desireable in some cases.

I think Dan might be in the camp that says a properly designed embedded
system won't need to swap.  And when I hear about how people do try and
swap on systems like this, I really start agreeing.  Maybe we could make
8xx just select SWAP=n? :)

TimeSys shipped their kernel with swapping fix as far as I know (Jason
plyed with it recently).

We'd better not assume what people try to do with their old 8xx's :)

quoted

quoted

I'm tempted to add a configuration option that is the complete
opposite of this and assumes are really embedded system.
Mark pages as always accessed, data pages as always dirty,
and you can eliminate lots of TLB faults in systems that are
fairly static.

It sounds tempting indeed, but should you really notice a performance increase 
out of this?

Compared to 8xx in 2.6 today?  Absolutely.

-- 
Tom Rini
http://gate.crashing.org/~trini/
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@ozlabs.org
https://ozlabs.org/mailman/listinfo/linuxppc-embedded