From: Jan Kratochvil <hidden> Date: 2007-11-26 22:41:46
Hi,
this testcase:
http://people.redhat.com/jkratoch/dabr-lost.c
reproduces a PPC DABR kernel bug. The variable `variable' should not get
modified as the thread modifying it should be caught by its DABR:
$ ./dabr-lost
TID 30914: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30915: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30916: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30914: hitting the variable
TID 30915: hitting the variable
TID 30916: hitting the variable
variable found = 30916, caught TID = 30914
TID 30916: DABR 0x10012a77
Variable got modified by a thread which has DABR still set!
At the `variable found =' line the parent ptracer found the TID thread 30916
wrote the value into the variable - despite it had DABR alrady set before.
As the behavior is dependent on the current weather I expect the scheduling
matters there.
It is important the target thread is in the `nanosleep' syscall. If you define
WORKAROUND_SET_DABR_IN_SYSCALL in the testcase it busyloops in the userland and
the bug gets no longer reproduced.
I got it reproduced on a utrace-patched kernel on dual-CPU Power5 and Roland
McGrath reported it reproduced on the vanilla upstream kernel on a Mac G5.
Regards,
Jan Kratochvil
modified as the thread modifying it should be caught by its DABR:
=20
$ ./dabr-lost
TID 30914: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30915: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30916: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30914: hitting the variable
TID 30915: hitting the variable
TID 30916: hitting the variable
variable found =3D 30916, caught TID =3D 30914
TID 30916: DABR 0x10012a77
Variable got modified by a thread which has DABR still set!
=20
This sounds like a bug recently reported by Uli Weigand. BenH
said he'd take a look, but it probably fell under the table.
The problem found by Uli is that on certain processors (Cell/B.E.
in his case), the DABRX register needs to be set in order for
the DABR to take effect.
Arnd <><
From: Jan Kratochvil <hidden> Date: 2007-11-28 08:59:19
On Tue, 27 Nov 2007 23:35:36 +0100, Arnd Bergmann wrote:
On Monday 26 November 2007, Jan Kratochvil wrote:
quoted
Hi,
this testcase:
http://people.redhat.com/jkratoch/dabr-lost.c
reproduces a PPC DABR kernel bug. The variable `variable' should not get
modified as the thread modifying it should be caught by its DABR:
$ ./dabr-lost
TID 30914: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30915: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30916: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30914: hitting the variable
TID 30915: hitting the variable
TID 30916: hitting the variable
variable found = 30916, caught TID = 30914
TID 30916: DABR 0x10012a77
Variable got modified by a thread which has DABR still set!
This sounds like a bug recently reported by Uli Weigand. BenH
said he'd take a look, but it probably fell under the table.
The problem found by Uli is that on certain processors (Cell/B.E.
in his case), the DABRX register needs to be set in order for
the DABR to take effect.
Please be aware DABR works fine if the same code runs just 1 (always) or
2 (sometimes) threads. It starts failing with too many threads running:
$ ./dabr-lost
TID 32725: DABR 0x1001279f NIP 0xfecf41c
TID 32726: DABR 0x1001279f NIP 0xfecf41c
TID 32725: hitting the variable
variable found = -1, caught TID = 32725
TID 32726: hitting the variable
variable found = -1, caught TID = 32726
The kernel bug did not get reproduced - increase THREADS.
As I did not find any code in that kernel touching DABRX its value should not
be dependent on the number of threads running.
Regards,
Lace
On Wednesday 28 November 2007, Jan Kratochvil wrote:
Please be aware DABR works fine if the same code runs just 1 (always) or
2 (sometimes) threads. =C2=A0It starts failing with too many threads runn=
ing:
=20
$ ./dabr-lost
TID 32725: DABR 0x1001279f NIP 0xfecf41c
TID 32726: DABR 0x1001279f NIP 0xfecf41c
TID 32725: hitting the variable
variable found =3D -1, caught TID =3D 32725
TID 32726: hitting the variable
variable found =3D -1, caught TID =3D 32726
The kernel bug did not get reproduced - increase THREADS.
=20
As I did not find any code in that kernel touching DABRX its value should=
not
be dependent on the number of threads running.
=20
Right, this is a different problem from the one reported by Uli.
=46rom what I can tell, your problem is that you set the DABR only
in one thread, so the other threads don't see it. DABR is saved
in the thread_struct, so setting it in one thread doesn't have
an impact on any other thread.
Arnd <><
From: Jan Kratochvil <hidden> Date: 2007-11-28 12:45:36
On Wed, 28 Nov 2007 13:28:48 +0100, Arnd Bergmann wrote:
On Wednesday 28 November 2007, Jan Kratochvil wrote:
quoted
Please be aware DABR works fine if the same code runs just 1 (always) or
2 (sometimes) threads. It starts failing with too many threads running:
$ ./dabr-lost
TID 32725: DABR 0x1001279f NIP 0xfecf41c
TID 32726: DABR 0x1001279f NIP 0xfecf41c
TID 32725: hitting the variable
variable found = -1, caught TID = 32725
TID 32726: hitting the variable
variable found = -1, caught TID = 32726
The kernel bug did not get reproduced - increase THREADS.
As I did not find any code in that kernel touching DABRX its value should not
be dependent on the number of threads running.
Right, this is a different problem from the one reported by Uli.
From what I can tell, your problem is that you set the DABR only
in one thread, so the other threads don't see it. DABR is saved
in the thread_struct, so setting it in one thread doesn't have
an impact on any other thread.
It even prints out above:
TID 32725: DABR 0x1001279f NIP 0xfecf41c
TID 32726: DABR 0x1001279f NIP 0xfecf41c
that it wrote DABR in both the threads and it has also successfully read it
back from each thread specifically (according to its thread-specific TID).
for (threadi = 0; threadi < THREADS; threadi++)
{
pid_t tid = thread[threadi];
setup (tid);
...
}
static void setup (pid_t tid)
{
...
l = ptrace (PTRACE_SET_DEBUGREG, tid, NULL, (void *) dabr);
...
}
Also if I would not set DABR specifically for each thread it would not work in
90% of cases for `THREADS == 2'. And it would not work for `THREADS == 4' if
they are busylooping (therefore not in a syscall).
TID 596: DABR 0x100127a7 NIP 0x10000dbc
TID 597: DABR 0x100127a7 NIP 0x10000db0
TID 598: DABR 0x100127a7 NIP 0x10000dac
TID 599: DABR 0x100127a7 NIP 0x10000dbc
TID 596: hitting the variable
variable found = -1, caught TID = 596
TID 599: hitting the variable
variable found = -1, caught TID = 599
TID 597: hitting the variable
variable found = -1, caught TID = 597
TID 598: hitting the variable
variable found = -1, caught TID = 598
The kernel bug got workarounded by WORKAROUND_SET_DABR_IN_SYSCALL.
(I found out now WORKAROUND_SET_DABR_IN_SYSCALL only reduces the probability of
the failure, it is not a 100% workaround of the problem in the testcase.)
There is some tricky kernel code around it but I did not try to debug it:
struct task_struct *__switch_to(struct task_struct *prev,
struct task_struct *new)
{
...
if (unlikely(__get_cpu_var(current_dabr) != new->thread.dabr)) {
set_dabr(new->thread.dabr);
__get_cpu_var(current_dabr) = new->thread.dabr;
}
...
}
Regards,
Jan
Hi,
this testcase:
http://people.redhat.com/jkratoch/dabr-lost.c
reproduces a PPC DABR kernel bug. The variable `variable' should not get
modified as the thread modifying it should be caught by its DABR:
$ ./dabr-lost
TID 30914: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30915: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30916: DABR 0x10012a77 NIP 0x80f6ebb318
TID 30914: hitting the variable
TID 30915: hitting the variable
TID 30916: hitting the variable
variable found = 30916, caught TID = 30914
TID 30916: DABR 0x10012a77
Variable got modified by a thread which has DABR still set!
This sounds like a bug recently reported by Uli Weigand. BenH
said he'd take a look, but it probably fell under the table.
The problem found by Uli is that on certain processors (Cell/B.E.
in his case), the DABRX register needs to be set in order for
the DABR to take effect.
Just as a note, the PS3's lv1_set_dabr(), which we used for
ppc_md.set_dabr sets up both the DABRX and DABR registers.
-Geoff
On Wednesday 28 November 2007 23:59:36 Geoff Levand wrote:
quoted
This sounds like a bug recently reported by Uli Weigand. BenH
said he'd take a look, but it probably fell under the table.
The problem found by Uli is that on certain processors (Cell/B.E.
in his case), the DABRX register needs to be set in order for
the DABR to take effect.
Just as a note, the PS3's lv1_set_dabr(), which we used for
ppc_md.set_dabr sets up both the DABRX and DABR registers.
Yes, I know. I tried it on the PS3 first and couldn't reproduce
the bug he saw on the blade.
Arnd <><
From: Luis Machado <hidden> Date: 2008-03-10 00:54:08
Yes, I know. I tried it on the PS3 first and couldn't reproduce
the bug he saw on the blade.
Arnd,
Do we have any news on this topic?
I've seen this happening quite often within GDB when using hardware
watchpoints on a shared variable in a threaded (7+ threads) binary.
Sometimes the watchpoint won't trigger, even though the monitored
variable's value was modified.
Appreciate your feedback.
Best regards,
--
Luis Machado
LoP Toolchain
Software Engineer
IBM Linux Technology Center
Yes, I know. I tried it on the PS3 first and couldn't reproduce
the bug he saw on the blade.
Arnd,
Do we have any news on this topic?
I've seen this happening quite often within GDB when using hardware
watchpoints on a shared variable in a threaded (7+ threads) binary.
Sometimes the watchpoint won't trigger, even though the monitored
variable's value was modified.
On the Blade DABRX had to be set additional to DABR. PS3 and Celleb
already did this. Uli Weigand found this back in November. I submitted
a patch for this which went into 2.6.25-rc4.
Can you please try again with rc4 ?
Gruß,
Jens
IBM Deutschland Entwicklung GmbH
Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Herbert Kircher
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
From: Luis Machado <hidden> Date: 2008-03-10 15:14:17
On the Blade DABRX had to be set additional to DABR. PS3 and Celleb
already did this. Uli Weigand found this back in November. I submitted
a patch for this which went into 2.6.25-rc4.
Can you please try again with rc4 ?
I will try it and will post the results back.
Thanks Jens.
Regards,
--
Luis Machado
Software Engineer
IBM Linux Technology Center
From: Luis Machado <hidden> Date: 2008-03-12 17:51:28
Hi,
On the Blade DABRX had to be set additional to DABR. PS3 and Celleb
already did this. Uli Weigand found this back in November. I submitted
a patch for this which went into 2.6.25-rc4.
Can you please try again with rc4 ?
Gruß,
Jens
Just to make sure, i tested the binary against the 2.6.25-rc4 kernel. It
still fails. So this is really an open bug for PPC.
--
Luis Machado
Software Engineer
IBM Linux Technology Center
Just to make sure, i tested the binary against the 2.6.25-rc4 kernel. It
still fails. So this is really an open bug for PPC.
On a Cell- or 970-based machine ?
Gruß,
Jens
IBM Deutschland Entwicklung GmbH
Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Herbert Kircher
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294