Ah, yes, I discovered ATOMIC_SYNC_FIX after I sent that, and have now
turned it on. That should certainly fix the atomic ops, however there
are quite a number of other places where the kernel uses stwcx., which
ATOMIX_SYNC_FIX doesn't fix - notably arch/ppc/kernel/bitops.c and
include/asm-ppc/bitops.h. As well as activating ATOMIX_SYNC_FIX I
tried inserting a sync before every other stwcx. that I could find,
and I haven't managed to get a process to lock up yet.
Just as an FYI, we also do this in glibc to be safe. We have never been
able to pin down a problem in userspace due to this bug, but we thought
it would be better safe then sorry until we can get definative proof
that the bug will not happen in userspace.
The following two files in glibc should be patched:
linuxthreads/sysdeps/powerpc/pt-machine.h
sysdeps/powerpc/atomicity.h
We may have an example of a problem in userspace caused by erratum 77.
We've been using a version of gcc3.0.2 running with your patched glibc, and
had a very mysterious crash that happened roughly once in 1000 minutes,
but only with a lucky build of our c++ app. The problem disappeared
if you looked at it in any way. Fortunately (?), we had a rack of 100
CPUs running the app, so the MTBF was 10 minutes.
It looks like the following file in gcc3 must be patched:
gcc-3.0.2/libstdc++-v3/config/cpu/powerpc/bits/atomicity.h
Here's my and Jan's attempt at a patch:
http://gcc.gnu.org/ml/libstdc++/2002-07/msg00152.html
Have your patches to glibc been contributed back to the mainline yet?
I'm looking carefully about how best to contribute my change;
I think it ought to be a multilib that is explicitly enabled.
- Dan
** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/
From: Mark Hatle <hidden> Date: 2002-07-20 15:14:26
The atonicity patches had not been submitted back to glibc due to there being
now way for me to show it was needed, and also that it ONLY affects the 405 CPU,
which isn't the main target of glibc.
We are currently working on revising our glibc patches to the current CVS
version, and if a new patch is required I'll make sure it gets posted here. I
really don't know the best way to handle this in a community glibc/gcc realm.
I'd almost like to wait and see what the GCC maintainers response is.
Specifically how they are going to accept the patch. Then we propose a similar
thing to the glibc folks, explain the problem and hope they accept the patch as
well. The biggest problem of course is that if we enable this for all PPCs
we're going to take a performance penalty for CPUs that don't have the problem.
As a community member I would hate to have to take the performance hit on my
750/74xx machines because of "some little embedded CPU". (Yes thats how I have
heard them described by others in similar situations.)
Any suggestions, anything I can do to help I am more then willing to just let me
know.
--Mark
dank@kegel.com wrote:
Mark Hatle [off-list ref] wrote on 18 Sep 2001:
quoted
David Gibson wrote:
quoted
Ah, yes, I discovered ATOMIC_SYNC_FIX after I sent that, and have now
turned it on. That should certainly fix the atomic ops, however there
are quite a number of other places where the kernel uses stwcx., which
ATOMIX_SYNC_FIX doesn't fix - notably arch/ppc/kernel/bitops.c and
include/asm-ppc/bitops.h. As well as activating ATOMIX_SYNC_FIX I
tried inserting a sync before every other stwcx. that I could find,
and I haven't managed to get a process to lock up yet.
Just as an FYI, we also do this in glibc to be safe. We have never been
able to pin down a problem in userspace due to this bug, but we thought
it would be better safe then sorry until we can get definative proof
that the bug will not happen in userspace.
The following two files in glibc should be patched:
linuxthreads/sysdeps/powerpc/pt-machine.h
sysdeps/powerpc/atomicity.h
We may have an example of a problem in userspace caused by erratum 77.
We've been using a version of gcc3.0.2 running with your patched glibc, and
had a very mysterious crash that happened roughly once in 1000 minutes,
but only with a lucky build of our c++ app. The problem disappeared
if you looked at it in any way. Fortunately (?), we had a rack of 100
CPUs running the app, so the MTBF was 10 minutes.
It looks like the following file in gcc3 must be patched:
gcc-3.0.2/libstdc++-v3/config/cpu/powerpc/bits/atomicity.h
Here's my and Jan's attempt at a patch:
http://gcc.gnu.org/ml/libstdc++/2002-07/msg00152.html
Have your patches to glibc been contributed back to the mainline yet?
I'm looking carefully about how best to contribute my change;
I think it ought to be a multilib that is explicitly enabled.
- Dan
The atonicity patches had not been submitted back to glibc due to there being
now way for me to show it was needed
That *is* a problem, isn't it? We have a test case, but not one we
can distribute. I'm trying to create a clean one that can be distributed,
but this bug is hard to trigger.
and also that it ONLY affects the 405 CPU,
which isn't the main target of glibc.
We are currently working on revising our glibc patches to the current CVS
version, and if a new patch is required I'll make sure it gets posted here. I
really don't know the best way to handle this in a community glibc/gcc realm.
I'd almost like to wait and see what the GCC maintainers response is.
Specifically how they are going to accept the patch. Then we propose a similar
thing to the glibc folks, explain the problem and hope they accept the patch as
well. The biggest problem of course is that if we enable this for all PPCs
we're going to take a performance penalty for CPUs that don't have the problem.
The patch can be made conditional, so that by default it affects nothing.
The libstdc++ mailing list has already blessed the part of the change
that affects them.
Here's how to do it:
1. Arrange for gcc to define __PPC405__ when appopriate by changing
gcc/config/rs6000/rs6000.h:
-%{mcpu=405: -D_ARCH_PPC} \
+%{mcpu=405: -D_ARCH_PPC -D__PPC405__} \
2. Surround the workaround with #ifdef __PPC405__. For instance,
in gcc-3.0.2/libstdc++-v3/config/cpu/powerpc/bits/atomicity.h:
+#ifdef __PPC405__
+#define __LIBSTDCPP_PPC405_ERR77_SYNC "sync \n\t"
+#else
+#define __LIBSTDCPP_PPC405_ERR77_SYNC
+#endif
+
static inline _Atomic_word
__attribute__ ((__unused__))
__exchange_and_add (volatile _Atomic_word* __mem, int __val)
4. Arrange to suppress generation of ppc405-specific libraries by default.
I have yet to do this, but it's probably a change to gcc/config-ml.in
to add an --enable-ppc405 option.
I could use help in two areas:
figuring out how to generate many interrupts on the ppc405 to make
a regression test more likely to run into the problem (see my previous
post to linuxppc-embedded), and
figuring out how gcc's MULTILIB_EXCEPTIONS parameter (I'll be posting
to the gcc list about that, so hopefully some expert there will
give me a clue).
Oh, and it'd be nice to hear what you think of my patch to gcc3, too.
It'd be nice to get ppc405 support in the gcc/glibc mainline, glad to hear
you're working on it too!
- Dan
** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/
From: Mark Hatle <hidden> Date: 2002-07-20 16:02:13
dank@kegel.com wrote:
Mark Hatle wrote:
quoted
The atonicity patches had not been submitted back to glibc due to there being
now way for me to show it was needed
That *is* a problem, isn't it? We have a test case, but not one we
can distribute. I'm trying to create a clean one that can be distributed,
but this bug is hard to trigger.
Well at least you have hit the problem, so at least it is shown to affect
someone. BTW what version of the kernel sources are you using?
The patch can be made conditional, so that by default it affects nothing.
The libstdc++ mailing list has already blessed the part of the change
that affects them.
Here's how to do it:
1. Arrange for gcc to define __PPC405__ when appopriate by changing
gcc/config/rs6000/rs6000.h:
-%{mcpu=405: -D_ARCH_PPC} \
+%{mcpu=405: -D_ARCH_PPC -D__PPC405__} \
2. Surround the workaround with #ifdef __PPC405__. For instance,
in gcc-3.0.2/libstdc++-v3/config/cpu/powerpc/bits/atomicity.h:
.....
Ya that all makes sense to me, and from the couple of threads I saw on the gcc
mailing list at least it seems to be acceptable to those folks as well.
As far as the multilib stuff goes I'll leave that to the gcc experts, I really
don't understand how the multilib stuff works. (We don't use it, we rebuild the
compiler to target the architecture itself instead of building one gcc that can
target all of the PPC archs.)
3. Arrange for ppc405-specific libraries to be generated. I'm
still working on this, but it's something like
...
4. Arrange to suppress generation of ppc405-specific libraries by default.
I have yet to do this, but it's probably a change to gcc/config-ml.in
to add an --enable-ppc405 option.
That is probably for the best.
I could use help in two areas:
figuring out how to generate many interrupts on the ppc405 to make
a regression test more likely to run into the problem (see my previous
post to linuxppc-embedded), and
One suggestion is to create a special kernel that has a hacked timer.c function
to generate a heck of a lot more timer interrupts.. But I don't know the
specifics on the kernel or the 405 CPU line to say if that is really practical
or not. The other folks in the embedded list would be better then me at that.
figuring out how gcc's MULTILIB_EXCEPTIONS parameter (I'll be posting
to the gcc list about that, so hopefully some expert there will
give me a clue).
It's very much voodoo magic, but unfortunatly I don't really understand it
either. Sparc, ARM and MIPS might be good examples as they seem to have many
more multilib modes then PPC. (Arm especially w/ the thumb modes).
Oh, and it'd be nice to hear what you think of my patch to gcc3, too.
Everything looks fine to me. It definatly looks correct. (In addition someone
should suggest that the libjava atomic operation be fixed as well.. not sure how
many people would use the GCC libjava stuff on 405 though....)
It'd be nice to get ppc405 support in the gcc/glibc mainline, glad to hear
you're working on it too!
I'm just not holding my breath for glibc support, I'm a bit pesimistic about the
PPC glibc maintainer(s). I've never had great luck submitting patches and
getting them accepted. (But it may just be me or the problems I always seem to
run into....)
If you have access to irc, log in to irc.openprojects.net and "#mklinux".
--Mark
** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/
We have a test case, but not one we can distribute....
Well at least you have hit the problem, so at least it is shown to affect
someone. BTW what version of the kernel sources are you using?
We're using 2.4.17 from linuxppc_2_4_devel as of late December 2001.
We do not yet have confirmation that we're running into erratum 77, but
it seems likely.
quoted
The patch can be made conditional, so that by default it affects nothing.
1. Arrange for gcc to define __PPC405__ when appopriate by changing
gcc/config/rs6000/rs6000.h:
-%{mcpu=405: -D_ARCH_PPC} \
+%{mcpu=405: -D_ARCH_PPC -D__PPC405__} \
2. Surround the workaround with #ifdef __PPC405__. For instance,
in gcc-3.0.2/libstdc++-v3/config/cpu/powerpc/bits/atomicity.h:
.....
Ya that all makes sense to me, and from the couple of threads I saw on the gcc
mailing list at least it seems to be acceptable to those folks as well.
As far as the multilib stuff goes I'll leave that to the gcc experts, I really
don't understand how the multilib stuff works. (We don't use it, we rebuild the
compiler to target the architecture itself instead of building one gcc that can
target all of the PPC archs.)
We don't use multilib, either -- well, that's not quite true. When we
build gcc for the ppc405, we grab the libraries from the 'nof' directory.
Turns out that's multilib in action. So all I'm doing is arranging
for a second directory, ppc405, to be generated next to nof. Like nof,
it has softfloat, but it also will have __PPC405__ turned on during library
build, which will trigger the workaround.
quoted
I could use help in two areas:
figuring out how to generate many interrupts on the ppc405 to make
a regression test more likely to run into the problem (see my previous
post to linuxppc-embedded), and
One suggestion is to create a special kernel that has a hacked timer.c function
to generate a heck of a lot more timer interrupts.. But I don't know the
specifics on the kernel or the 405 CPU line to say if that is really practical
or not. The other folks in the embedded list would be better then me at that.
OK. Can some ppc405 expert speak up?
quoted
Oh, and it'd be nice to hear what you think of my patch to gcc3, too.
Everything looks fine to me. It definatly looks correct. (In addition someone
should suggest that the libjava atomic operation be fixed as well.. not sure how
many people would use the GCC libjava stuff on 405 though....)
Thanks. If I get a chance, I'll look at libjava/sysdep/powerpc/locks.h, too.
quoted
It'd be nice to get ppc405 support in the gcc/glibc mainline, glad to hear
you're working on it too!
I'm just not holding my breath for glibc support ...
glibc is notoriously difficult to contribute to. I would be satisfied
with a separately maintained patch that applies to the latest glibc,
but I am optimistic that if I dot all my i's and cross my t's, I
can get a patch accepted.
- Dan
** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/
The atonicity patches had not been submitted back to glibc due to there being
now way for me to show it was needed, and also that it ONLY affects the 405 CPU,
which isn't the main target of glibc.
We are currently working on revising our glibc patches to the current CVS
version, and if a new patch is required I'll make sure it gets posted here. I
really don't know the best way to handle this in a community glibc/gcc realm.
I'd almost like to wait and see what the GCC maintainers response is.
Specifically how they are going to accept the patch. Then we propose a similar
thing to the glibc folks, explain the problem and hope they accept the patch as
well.
From: Mark Hatle <hidden> Date: 2002-07-23 13:10:18
dank@kegel.com wrote:
Mark Hatle wrote:
quoted
The atonicity patches had not been submitted back to glibc due to there being
now way for me to show it was needed, and also that it ONLY affects the 405 CPU,
which isn't the main target of glibc.
We are currently working on revising our glibc patches to the current CVS
version, and if a new patch is required I'll make sure it gets posted here. I
really don't know the best way to handle this in a community glibc/gcc realm.
I'd almost like to wait and see what the GCC maintainers response is.
Specifically how they are going to accept the patch. Then we propose a similar
thing to the glibc folks, explain the problem and hope they accept the patch as
well.
Mark,
I've rediffed your glibc patch and made it conditional on defined(__PPC405__);
result at http://www.kegel.com/xgcc3/glibc-2.2.5-ppc405erratum77.patch
I haven't tested it yet, but something like that should make the maintainers happy.
What do you think?
Sounds good.. (as I said before we're in the process of re doing our diff for
2.2.5 as well, so I'll compare them and make sure neither of us missed
anything. We havn't added the __PPC405__ bit yet, but planned on doing so
soon.)