pkeys on POWER: Default AMR, UAMOR values

I'm working on adding POWER pkeys support to glibc.  The coding work is 
done, but I'm faced with some test suite failures.

Unlike the default x86 configuration, on POWER, existing threads have 
full access to newly allocated keys.

Or, more precisely, in this scenario:

* Thread A launches thread B
* Thread B waits
* Thread A allocations a protection key with pkey_alloc
* Thread A applies the key to a page
* Thread A signals thread B
* Thread B starts to run and accesses the page

Then at the end, the access will be granted.

I hope it's not too late to change this to denied access.

Furthermore, I think the UAMOR value is wrong as well because it 
prevents thread B at the end to set the AMR register.  In particular, if 
I do this

* … (as before)
* Thread A signals thread B
* Thread B sets the access rights for the key to PKEY_DISABLE_ACCESS
* Thread B reads the current access rights for the key

then it still gets 0 (all access permitted) because the original UAMOR 
value inherited from thread A prior to the key allocation masks out the 
access right update for the newly allocated key.

Thanks,
Florian

On Fri, May 18, 2018 at 6:17 AM Florian Weimer [off-list ref] wrote:

I'm working on adding POWER pkeys support to glibc.  The coding work is
done, but I'm faced with some test suite failures.

Unlike the default x86 configuration, on POWER, existing threads have
full access to newly allocated keys.

Or, more precisely, in this scenario:

* Thread A launches thread B
* Thread B waits
* Thread A allocations a protection key with pkey_alloc
* Thread A applies the key to a page
* Thread A signals thread B
* Thread B starts to run and accesses the page

Then at the end, the access will be granted.

I hope it's not too late to change this to denied access.

Furthermore, I think the UAMOR value is wrong as well because it
prevents thread B at the end to set the AMR register.  In particular, if
I do this

* =E2=80=A6 (as before)
* Thread A signals thread B
* Thread B sets the access rights for the key to PKEY_DISABLE_ACCESS
* Thread B reads the current access rights for the key

then it still gets 0 (all access permitted) because the original UAMOR
value inherited from thread A prior to the key allocation masks out the
access right update for the newly allocated key.

This type of issue is why I think that a good protection key ISA would not
have a usermode read-the-whole-register or write-the-whole-register
operation at all.  It's still not clear to me that there is any good
kernel-mode solution.  But at least x86 defaults to deny-everything, which
is more annoying but considerably safer than POWER's behavior.

--Andy

On Fri, May 18, 2018 at 03:17:14PM +0200, Florian Weimer wrote:

I'm working on adding POWER pkeys support to glibc.  The coding work
is done, but I'm faced with some test suite failures.

Unlike the default x86 configuration, on POWER, existing threads
have full access to newly allocated keys.

Or, more precisely, in this scenario:

* Thread A launches thread B
* Thread B waits
* Thread A allocations a protection key with pkey_alloc
* Thread A applies the key to a page
* Thread A signals thread B
* Thread B starts to run and accesses the page

Then at the end, the access will be granted.

I hope it's not too late to change this to denied access.

Furthermore, I think the UAMOR value is wrong as well because it
prevents thread B at the end to set the AMR register.  In
particular, if I do this

* … (as before)
* Thread A signals thread B
* Thread B sets the access rights for the key to PKEY_DISABLE_ACCESS
* Thread B reads the current access rights for the key

then it still gets 0 (all access permitted) because the original
UAMOR value inherited from thread A prior to the key allocation
masks out the access right update for the newly allocated key.

Florian, is the behavior on x86 any different? A key allocated in the
context off one thread is not meaningful in the context of any other
thread. 

Since thread B was created prior to the creation of the key, and the key
was created in the context of thread A, thread B neither inherits the
key nor its permissions. Atleast that is how the semantics are supposed
to work as per the man page.

man 7 pkey 

" Applications  using  threads  and  protection  keys  should
be especially careful.  Threads inherit the protection key rights of the
parent at the time of the clone(2), system call.  Applications should
either ensure that their own permissions are appropriate for child
threads at the time when clone(2) is  called,  or ensure that each child
thread can perform its own initialization of protection key rights."


RP

On Fri, May 18, 2018 at 10:45 AM Ram Pai [off-list ref] wrote:

On Fri, May 18, 2018 at 03:17:14PM +0200, Florian Weimer wrote:

quoted

I'm working on adding POWER pkeys support to glibc.  The coding work
is done, but I'm faced with some test suite failures.

Unlike the default x86 configuration, on POWER, existing threads
have full access to newly allocated keys.

Or, more precisely, in this scenario:

* Thread A launches thread B
* Thread B waits
* Thread A allocations a protection key with pkey_alloc
* Thread A applies the key to a page
* Thread A signals thread B
* Thread B starts to run and accesses the page

Then at the end, the access will be granted.

I hope it's not too late to change this to denied access.

Furthermore, I think the UAMOR value is wrong as well because it
prevents thread B at the end to set the AMR register.  In
particular, if I do this

* =E2=80=A6 (as before)
* Thread A signals thread B
* Thread B sets the access rights for the key to PKEY_DISABLE_ACCESS
* Thread B reads the current access rights for the key

then it still gets 0 (all access permitted) because the original
UAMOR value inherited from thread A prior to the key allocation
masks out the access right update for the newly allocated key.

Florian, is the behavior on x86 any different? A key allocated in the
context off one thread is not meaningful in the context of any other
thread.

The difference is that x86 starts out with deny-all instead of allow-all.
The POWER semantics make it very hard for a multithreaded program to
meaningfully use protection keys to prevent accidental access to important
memory.

On 05/18/2018 07:44 PM, Ram Pai wrote:

Florian, is the behavior on x86 any different? A key allocated in the
context off one thread is not meaningful in the context of any other
thread.

Since thread B was created prior to the creation of the key, and the key
was created in the context of thread A, thread B neither inherits the
key nor its permissions. Atleast that is how the semantics are supposed
to work as per the man page.

man 7 pkey

" Applications  using  threads  and  protection  keys  should
be especially careful.  Threads inherit the protection key rights of the
parent at the time of the clone(2), system call.  Applications should
either ensure that their own permissions are appropriate for child
threads at the time when clone(2) is  called,  or ensure that each child
thread can perform its own initialization of protection key rights."

I reported two separate issues (actually three, but the execve bug is in 
a separate issue).  The default, and the write restrictions.

The default is just a difference to x86 (however, x86 can be booted with 
init_pkru=0 and behaves the same way, but we're probably going to remove 
that).

The POWER implementation has the additional wrinkle that threads 
launched early, before key allocation, can never change access rights 
because they inherited not just the access rights, but also the access 
rights access mask.  This is different from x86, where all threads can 
freely update access rights, and contradicts the behavior in the manpage 
which says that “each child thread can perform its own initialization of 
protection key rights”.  It can't do that if it is launched before key 
allocation, which is not the right behavior IMO.

Thanks,
Florian

On 05/18/2018 09:39 PM, Andy Lutomirski wrote:

The difference is that x86 starts out with deny-all instead of allow-all.
The POWER semantics make it very hard for a multithreaded program to
meaningfully use protection keys to prevent accidental access to important
memory.

And you can change access rights for unallocated keys (unallocated at 
thread start time, allocated later) on x86.  I have extended the 
misc/tst-pkeys test to verify that, and it passes on x86, but not on 
POWER, where the access rights are stuck.

I believe this is due to an incorrect UAMOR setting.

Thanks,
Florian

On Fri, May 18, 2018 at 11:13:30PM +0200, Florian Weimer wrote:

On 05/18/2018 09:39 PM, Andy Lutomirski wrote:

quoted

The difference is that x86 starts out with deny-all instead of allow-all.

Ah!. this explains the discrepency. But still does not explain one
thing.. see below.

quoted

The POWER semantics make it very hard for a multithreaded program to
meaningfully use protection keys to prevent accidental access to important
memory.

And you can change access rights for unallocated keys (unallocated
at thread start time, allocated later) on x86.  I have extended the
misc/tst-pkeys test to verify that, and it passes on x86, but not on
POWER, where the access rights are stuck.

This is something I do not understand. How can a thread change permissions
on a key, that is not even allocated in the first place. Do you consider a key
allocated in some other thread's context, as allocated in this threads
context? If not, does that mean -- On x86, you can activate a key just
by changing its permission?


RP

On 05/19/2018 02:52 AM, Ram Pai wrote:

quoted

quoted

The POWER semantics make it very hard for a multithreaded program to
meaningfully use protection keys to prevent accidental access to important
memory.

And you can change access rights for unallocated keys (unallocated
at thread start time, allocated later) on x86.  I have extended the
misc/tst-pkeys test to verify that, and it passes on x86, but not on
POWER, where the access rights are stuck.

This is something I do not understand. How can a thread change permissions
on a key, that is not even allocated in the first place.

It was allocated by another thread, and there is synchronization so that 
the allocation happens before the change in access rights.

Do you consider a key
allocated in some other thread's context, as allocated in this threads
context?

Yes, x86 does that.

If not, does that mean -- On x86, you can activate a key just
by changing its permission?

This also true on x86, but just an artifact of the implementation.  You 
are supposed to call pkey_alloc before changing the flag.

Thanks,
Florian