Hi all
I have been playing with the copy_page() function in arch/ppc/kernel/misc.S
and gained about 30% speed up for my mpc860, rev D4 MHz.
This is what i did:
- Use dcbz on 8xx but clear ahead one cache line(performance is really crappy
if I don't clear ahead). This is the biggest improvement.
- Use prefetch for 8xx as well.
I know that dcbz is buggy for some 8xx CPUs but I don't know which ones.
For me works just fine, except in copy_tofrom_user(don't know why).
I would like to get some feedback & test results both for 8xx and non 8xx.
Please include exact CPU and revision.
Thanks
Jocke
_GLOBAL(copy_page)
addi r3,r3,-4
addi r4,r4,-4
li r5,4
#if MAX_COPY_PREFETCH > 1
/* This will prefetch past end of page, does not seem to be a problem? */
li r0,MAX_COPY_PREFETCH
li r11,4
mtctr r0
11: dcbt r11,r4
addi r11,r11,L1_CACHE_LINE_SIZE
bdnz 11b
#else /* MAX_L1_COPY_PREFETCH == 1 */
dcbt r5,r4
li r11,L1_CACHE_LINE_SIZE+4
#endif /* MAX_L1_COPY_PREFETCH */
dcbz r5,r3 /* older 8xx CPUs may have buggy dcbz instructions, if so try "dcbt r5,r3" instead */
addi r5,r5,L1_CACHE_LINE_SIZE
li r0,4096/L1_CACHE_LINE_SIZE-1 /* All, but the last cache line of data due dcbz below */
mtctr r0
1:
dcbt r11,r4
dcbz r5,r3 /* zero the cache line after the one that is beeing copied
* older 8xx CPUs may have buggy dcbz instructions, if so try "dcbt r5,r3" instead */
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 32
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 64
COPY_16_BYTES
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 128
COPY_16_BYTES
COPY_16_BYTES
COPY_16_BYTES
COPY_16_BYTES
#endif
#endif
#endif
bdnz 1b
/* Copy the last cache line of data */
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 32
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 64
COPY_16_BYTES
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 128
COPY_16_BYTES
COPY_16_BYTES
COPY_16_BYTES
COPY_16_BYTES
#endif
#endif
#endif
blr
** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/
Hi all
I have been playing with the copy_page() function in arch/ppc/kernel/misc.S
and gained about 30% speed up for my mpc860, rev D4 MHz.
This is what i did:
- Use dcbz on 8xx but clear ahead one cache line(performance is really crappy
if I don't clear ahead). This is the biggest improvement.
- Use prefetch for 8xx as well.
I know that dcbz is buggy for some 8xx CPUs but I don't know which ones.
For me works just fine, except in copy_tofrom_user(don't know why).
hmm, I made two versions of copy_tofrom_user(), copy_from_user() and copy_to_user()
and modified asm/uaccess.h to reflect this.
Then I modified copy_from_user() to use dcbz on the destination area.
Booted and started our app and it works just fine!
Jocke
** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/
Forget the copy_page below. I used a non cache aligned buffer :-(
However if I enable the use of "dcbz" and remove "dcbt" in the
orginal copy_page() and use a cache aligned test buffer,
I still get a speedup of 30% or more on my mpc860 board.
I think a new CONFIG option is apropiate where one can turn
on the use of "dcbz" for 8xx. OK?
Jocke
Hi all
I have been playing with the copy_page() function in arch/ppc/kernel/misc.S
and gained about 30% speed up for my mpc860, rev D4 MHz.
This is what i did:
- Use dcbz on 8xx but clear ahead one cache line(performance is really crappy
if I don't clear ahead). This is the biggest improvement.
- Use prefetch for 8xx as well.
I know that dcbz is buggy for some 8xx CPUs but I don't know which ones.
For me works just fine, except in copy_tofrom_user(don't know why).
I would like to get some feedback & test results both for 8xx and non 8xx.
Please include exact CPU and revision.
Thanks
Jocke
_GLOBAL(copy_page)
addi r3,r3,-4
addi r4,r4,-4
li r5,4
#if MAX_COPY_PREFETCH > 1
/* This will prefetch past end of page, does not seem to be a problem? */
li r0,MAX_COPY_PREFETCH
li r11,4
mtctr r0
11: dcbt r11,r4
addi r11,r11,L1_CACHE_LINE_SIZE
bdnz 11b
#else /* MAX_L1_COPY_PREFETCH == 1 */
dcbt r5,r4
li r11,L1_CACHE_LINE_SIZE+4
#endif /* MAX_L1_COPY_PREFETCH */
dcbz r5,r3 /* older 8xx CPUs may have buggy dcbz instructions, if so try "dcbt r5,r3" instead */
addi r5,r5,L1_CACHE_LINE_SIZE
li r0,4096/L1_CACHE_LINE_SIZE-1 /* All, but the last cache line of data due dcbz below */
mtctr r0
1:
dcbt r11,r4
dcbz r5,r3 /* zero the cache line after the one that is beeing copied
* older 8xx CPUs may have buggy dcbz instructions, if so try "dcbt r5,r3" instead */
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 32
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 64
COPY_16_BYTES
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 128
COPY_16_BYTES
COPY_16_BYTES
COPY_16_BYTES
COPY_16_BYTES
#endif
#endif
#endif
bdnz 1b
/* Copy the last cache line of data */
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 32
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 64
COPY_16_BYTES
COPY_16_BYTES
#if L1_CACHE_LINE_SIZE >= 128
COPY_16_BYTES
COPY_16_BYTES
COPY_16_BYTES
COPY_16_BYTES
#endif
#endif
#endif
blr
From: Dan Malek <hidden> Date: 2003-03-03 21:28:37
Joakim Tjernlund wrote:
I have been playing with the copy_page() function in arch/ppc/kernel/misc.S
and gained about 30% speed up for my mpc860, rev D4 MHz.
Have you found the discussion in linuxppc-dev about the work Paul has done
on this in general for PowerPC? It may help avoid repeating some work and
provide some guidance.....
And don't forget....many applications aren't heavily 'copy-centric' and it
may be beneficial to not blow away the caches in those cases. That is, if
you apply systems engineering methods to your testing instead of just focusing
on such a low level detail, you may discover you are wasting your time and
from an overall system application you may be providing little benefit or
even a overall degradation in system performance.
Thanks.
-- Dan
** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/
I have been playing with the copy_page() function in arch/ppc/kernel/misc.S
and gained about 30% speed up for my mpc860, rev D4 MHz.
Have you found the discussion in linuxppc-dev about the work Paul has done
on this in general for PowerPC? It may help avoid repeating some work and
provide some guidance.....
I have searched but I did not find anything conclusive. Pointers?
And don't forget....many applications aren't heavily 'copy-centric' and it
may be beneficial to not blow away the caches in those cases. That is, if
If you are referring to the copy_page() that I attached in my first mail, then
yes it uses more icache, but if you have seen my later post where I took it
back and stated that just enabling dcbz in the existing version of copy_page()
would give the same speed up, I don't follow you. How am I wasting caches?
In the end I would like to modify copy_tofrom_user() so that dcbz is used on kernel space
addresses but not on user space without adding a lot of code. Ideas welcome.
you apply systems engineering methods to your testing instead of just focusing
on such a low level detail, you may discover you are wasting your time and
from an overall system application you may be providing little benefit or
even a overall degradation in system performance.
I am just trying make 8xx perform a little better and I focus on areas I know something
about such as crc32, the enet.c driver and in this case various memory copy stuff. Hopefully
the end result will be useful to me and others.
Jocke
** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/
From: Paul Mackerras <hidden> Date: 2003-03-04 00:19:11
Dan Malek writes:
And don't forget....many applications aren't heavily 'copy-centric' and it
may be beneficial to not blow away the caches in those cases. That is, if
Using dcbz on the destination won't blow away the caches any more than
doing the copy without dcbz would anyway.
The thing you have to be careful of when using dcbz, particularly when
you are dbcz'ing one or more cache lines ahead, is that you only dcbz
cache lines that are completely contained within the destination area.
That introduces extra complexity and makes the code bigger, so you
have to be careful that you don't make small copies slower.
I did some measurements once and found that almost all of the copies
in the kernel (memcpy and copy_tofrom_user) were either relatively
small, i.e. less than 256 bytes, or were page-sized and page-aligned.
In the optimized copy routines I did in the ppc64 kernel.
Paul.
** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/