PowerPC reservations

Hi,

Can someone please help me understand how the memory reservations in PowerPC actually work. Let's just assume uniprocessor with a pre-emptive scheduler, and take a text-book example of an atomic increment case, which is also frequently used in e.g. the Linux kernel. With two atomic operations, everything seems to be just fine. But how about with three concurrent threads of execution?

From the following code, assume r3 contains the same address for each incrementing operation. If the first atomic increment is pre-empted  in the middle, execution then jumps to the second increment (by the scheduler). The second increment runs through and succeeds, and continues straight to the third increment. Then it is again pre-empted in the middle, execution returning to the first increment. Now the processor has the reservation with the correct address, and the first increment succeeds when still holding the original input value. The first and the second increment thus write the same value in memory. After the first increment, the scheduler again continues the third increment, which doesn't succeed a first, but the second round succeeds. However, the value in the address pointed by r3 was not increased by three, but by two.

Am I just not getting how this is really supposed to work? Are there still some other constructs in use to prevent this, e.g. extra stwcx. instructions when changing the thread of execution?

I'm also wondering why the architecture specifically defines the stwcx. instruction to have, well, undefined behavior in case the reservation address differs from the address of the previous lwarx...

1:      lwarx r6, r0, r3
       addi r6, r6, 1
       stwcx. r6, r0, r3
       bne- 1b

.....

2:      lwarx r7, r0, r3
       addi r7, r7, 1
       stwcx. r7, r0, r3
       bne- 2b

3:      lwarx r8, r0, r3
       addi r8, r8, 1
       stwcx. r8, r0, r3
       bne- 3b

Kalle Pokki wrote:

Am I just not getting how this is really supposed to work? Are there still some other constructs in use to prevent this, e.g. extra stwcx. instructions when changing the thread of execution?

Yes. When a context switch occurs, there's a stcwx. to clear any reservations from the previous thread. When you enter another thread from a context switch, you enter it with no reservations pending.

So you might have something that looks like this:

thread 1 -> lwarx w/ successful reservation
context switch -> stwcx.. to garbage location to clear any reservations
thread 2 -> stwcx. fails since there's no reservation pending
context switch -> stwcx. to garbage location to clear any reservations
thread 1 -> stwcx. fails since there's no reservation pending
thread 1 -> loops to try lwarx again

So the context switcher protects a reservation from "leaking" into another thread. In that sense, you can act as though lwarx/stwcx. pairs belong to a single thread.

The other important thing to remember is that there can only be one outstanding reservation. Thus, lwarx and stcwx. instructions have to be "paired" or you're going to get unexpected results.