From: Andy Lutomirski <luto@kernel.org> Date: 2020-07-30 20:54:22
On Thu, Jul 30, 2020 at 7:24 AM Madhavan T. Venkataraman
[off-list ref] wrote:
Sorry for the delay. I just wanted to think about this a little.
In this email, I will respond to your first suggestion. I will
respond to the rest in separate emails if that is alright with
you.
On 7/28/20 12:31 PM, Andy Lutomirski wrote:
On Jul 28, 2020, at 6:11 AM, madvenka@linux.microsoft.com wrote:
From: "Madhavan T. Venkataraman" [off-list ref]
The kernel creates the trampoline mapping without any permissions. When
the trampoline is executed by user code, a page fault happens and the
kernel gets control. The kernel recognizes that this is a trampoline
invocation. It sets up the user registers based on the specified
register context, and/or pushes values on the user stack based on the
specified stack context, and sets the user PC to the requested target
PC. When the kernel returns, execution continues at the target PC.
So, the kernel does the work of the trampoline on behalf of the
application.
This is quite clever, but now I’m wondering just how much kernel help
is really needed. In your series, the trampoline is an non-executable
page. I can think of at least two alternative approaches, and I'd
like to know the pros and cons.
1. Entirely userspace: a return trampoline would be something like:
1:
pushq %rax
pushq %rbc
pushq %rcx
...
pushq %r15
movq %rsp, %rdi # pointer to saved regs
leaq 1b(%rip), %rsi # pointer to the trampoline itself
callq trampoline_handler # see below
You would fill a page with a bunch of these, possibly compacted to get
more per page, and then you would remap as many copies as needed. The
'callq trampoline_handler' part would need to be a bit clever to make
it continue to work despite this remapping. This will be *much*
faster than trampfd. How much of your use case would it cover? For
the inverse, it's not too hard to write a bit of asm to set all
registers and jump somewhere.
Let me state what I have understood about this suggestion. Correct me if
I get anything wrong. If you don't mind, I will also take the liberty
of generalizing and paraphrasing your suggestion.
The goal is to create two page mappings that are adjacent to each other:
- a code page that contains template code for a trampoline. Since the
template code would tend to be small in size, pack as many of them
as possible within a page to conserve memory. In other words, create
an array of the template code fragments. Each element in the array
would be used for one trampoline instance.
- a data page that contains an array of data elements. Corresponding
to each code element in the code page, there would be a data element
in the data page that would contain data that is specific to a
trampoline instance.
- Code will access data using PC-relative addressing.
The management of the code pages and allocation for each trampoline
instance would all be done in user space.
Is this the general idea?
Yes.
Creating a code page
--------------------
We can do this in one of the following ways:
- Allocate a writable page at run time, write the template code into
the page and have execute permissions on the page.
- Allocate a writable page at run time, write the template code into
the page and remap the page with just execute permissions.
- Allocate a writable page at run time, write the template code into
the page, write the page into a temporary file and map the file with
execute permissions.
- Include the template code in a code page at build time itself and
just remap the code page each time you need a code page.
This latter part shouldn't need any special permissions as far as I know.
Pros and Cons
-------------
As long as the OS provides the functionality to do this and the security
subsystem in the OS allows the actions, this is totally feasible. If not,
we need something like trampfd.
As Floren mentioned, libffi does implement something like this for MACH.
In fact, in my libffi changes, I use trampfd only after all the other methods
have failed because of security settings.
But the above approach only solves the problem for this simple type of
trampoline. It does not provide a framework for addressing more complex types
or even other forms of dynamic code.
Also, each application would need to implement this solution for itself
as opposed to relying on one implementation provided by the kernel.
I would argue this is a benefit. If the whole implementation is in
userspace, there is no ABI compatibility issue. The user program
contains the trampoline code and the code that uses it.
Trampfd-based solution
----------------------
I outlined an enhancement to trampfd in a response to David Laight. In this
enhancement, the kernel is the one that would set up the code page.
The kernel would call an arch-specific support function to generate the
code required to load registers, push values on the stack and jump to a PC
for a trampoline instance based on its current context. The trampoline
instance data could be baked into the code.
My initial idea was to only have one trampoline instance per page. But I
think I can implement multiple instances per page. I just have to manage
the trampfd file private data and VMA private data accordingly to map an
element in a code page to its trampoline object.
The two approaches are similar except for the detail about who sets up
and manages the trampoline pages. In both approaches, the performance problem
is addressed. But trampfd can be used even when security settings are
restrictive.
Is my solution acceptable?
Perhaps. In general, before adding a new ABI to the kernel, it's nice
to understand how it's better than doing the same thing in userspace.
Saying that it's easier for user code to work with if it's in the
kernel isn't necessarily an adequate justification.
Why would remapping two pages of actual application text ever fail?
From: Madhavan T. Venkataraman <hidden> Date: 2020-07-31 17:14:00
On 7/30/20 3:54 PM, Andy Lutomirski wrote:
On Thu, Jul 30, 2020 at 7:24 AM Madhavan T. Venkataraman
[off-list ref] wrote:
quoted
...
Creating a code page
--------------------
We can do this in one of the following ways:
- Allocate a writable page at run time, write the template code into
the page and have execute permissions on the page.
- Allocate a writable page at run time, write the template code into
the page and remap the page with just execute permissions.
- Allocate a writable page at run time, write the template code into
the page, write the page into a temporary file and map the file with
execute permissions.
- Include the template code in a code page at build time itself and
just remap the code page each time you need a code page.
This latter part shouldn't need any special permissions as far as I know.
Agreed.
quoted
Pros and Cons
-------------
As long as the OS provides the functionality to do this and the security
subsystem in the OS allows the actions, this is totally feasible. If not,
we need something like trampfd.
As Floren mentioned, libffi does implement something like this for MACH.
In fact, in my libffi changes, I use trampfd only after all the other methods
have failed because of security settings.
But the above approach only solves the problem for this simple type of
trampoline. It does not provide a framework for addressing more complex types
or even other forms of dynamic code.
Also, each application would need to implement this solution for itself
as opposed to relying on one implementation provided by the kernel.
I would argue this is a benefit. If the whole implementation is in
userspace, there is no ABI compatibility issue. The user program
contains the trampoline code and the code that uses it.
The current trampfd implementation also does not have an ABI issue.
ABI details are to be handled in user land. In the case of libffi, they
are. Trampfd only addresses the trampoline required to jump to the
ABI handler.
quoted
Trampfd-based solution
----------------------
I outlined an enhancement to trampfd in a response to David Laight. In this
enhancement, the kernel is the one that would set up the code page.
The kernel would call an arch-specific support function to generate the
code required to load registers, push values on the stack and jump to a PC
for a trampoline instance based on its current context. The trampoline
instance data could be baked into the code.
My initial idea was to only have one trampoline instance per page. But I
think I can implement multiple instances per page. I just have to manage
the trampfd file private data and VMA private data accordingly to map an
element in a code page to its trampoline object.
The two approaches are similar except for the detail about who sets up
and manages the trampoline pages. In both approaches, the performance problem
is addressed. But trampfd can be used even when security settings are
restrictive.
Is my solution acceptable?
Perhaps. In general, before adding a new ABI to the kernel, it's nice
to understand how it's better than doing the same thing in userspace.
Saying that it's easier for user code to work with if it's in the
kernel isn't necessarily an adequate justification.
Fair enough.
Dealing with multiple architectures
-----------------------------------------------
One good reason to use trampfd is multiple architecture support. The
trampoline table in a code page approach is neat. I don't deny that at
all. But my question is - can it be used in all cases?
It requires PC-relative data references. I have not worked on all architectures.
So, I need to study this. But do all ISAs support PC-relative data references?
Even in an ISA that supports it, there would be a maximum supported offset
from the current PC that can be reached for a data reference. That maximum
needs to be at least the size of a base page in the architecture. This is because
the code page and the data page need to be separate for security reasons.
Do all ISAs support a sufficiently large offset?
When the kernel generates the code for a trampoline, it can hard code data values
in the generated code itself so it does not need PC-relative data referencing.
And, for ISAs that do support the large offset, we do have to implement and
maintain the code page stuff for different ISAs for each application and library
if we did not use trampfd.
If you look at the libffi reference patch that I have linked in the cover letter,
I have added functions in common code that wrap trampfd calls. From architecture
specific code, there is just one function call to one of those wrapper functions
to set the register context for the trampoline. This is a very small C code change
in each architecture. So, support can be extended to all architectures without
exception easily.
Runtime generated trampolines
-------------------------------------------
libffi trampolines are simple. But there may be many cases out there where the
trampoline code cannot be statically defined at build time. It may have to be
generated at runtime. For this, we will need trampfd.
Security
-----------
With the user level trampoline table approach, the data part of the trampoline table
can be hacked by an attacker if an application has a vulnerability. Specifically, the
target PC can be altered to some arbitrary location. Trampfd implements an
"Allowed PCS" context. In the libffi changes, I have created a read-only array of
all ABI handlers used in closures for each architecture. This read-only array
can be used to restrict the PC values for libffi trampolines to prevent hacking.
To generalize, we can implement security rules/features if the trampoline
object is in the kernel.
Standardization
---------------------
Trampfd is a framework that can be used to implement multiple things. May be,
a few of those things can also be implemented in user land itself. But I think having
just one mechanism to execute dynamic code objects is preferable to having
multiple mechanisms not standardized across all applications.
As an example, let us say that I am able to implement support for JIT code. Let us
say that an interpreter uses libffi to execute a generated function. The interpreter
would use trampfd for the JIT code object and get an address. Then, it would pass
that to libffi which would then use trampfd for the trampoline. So, trampfd based
code objects can be chained.
Why would remapping two pages of actual application text ever fail?
Remapping a page may not be available on all OSes. However, that is not a problem
for the code page approach. One can always memory map the code page from the
binary file directly. So, yes, this would not fail.
Madhavan
From: Mark Rutland <mark.rutland@arm.com> Date: 2020-07-31 18:31:57
On Fri, Jul 31, 2020 at 12:13:49PM -0500, Madhavan T. Venkataraman wrote:
On 7/30/20 3:54 PM, Andy Lutomirski wrote:
quoted
On Thu, Jul 30, 2020 at 7:24 AM Madhavan T. Venkataraman
[off-list ref] wrote:
Dealing with multiple architectures
-----------------------------------------------
One good reason to use trampfd is multiple architecture support. The
trampoline table in a code page approach is neat. I don't deny that at
all. But my question is - can it be used in all cases?
It requires PC-relative data references. I have not worked on all architectures.
So, I need to study this. But do all ISAs support PC-relative data references?
Not all do, but pretty much any recent ISA will as it's a practical
necessity for fast position-independent code.
Even in an ISA that supports it, there would be a maximum supported offset
from the current PC that can be reached for a data reference. That maximum
needs to be at least the size of a base page in the architecture. This is because
the code page and the data page need to be separate for security reasons.
Do all ISAs support a sufficiently large offset?
ISAs with pc-relative addessing can usually generate PC-relative
addresses into a GPR, from which they can apply an arbitrarily large
offset.
When the kernel generates the code for a trampoline, it can hard code data values
in the generated code itself so it does not need PC-relative data referencing.
And, for ISAs that do support the large offset, we do have to implement and
maintain the code page stuff for different ISAs for each application and library
if we did not use trampfd.
Trampoline code is architecture specific today, so I don't see that as a
major issue. Common structural bits can probably be shared even if the
specifid machine code cannot.
[...]
Security
-----------
With the user level trampoline table approach, the data part of the trampoline table
can be hacked by an attacker if an application has a vulnerability. Specifically, the
target PC can be altered to some arbitrary location. Trampfd implements an
"Allowed PCS" context. In the libffi changes, I have created a read-only array of
all ABI handlers used in closures for each architecture. This read-only array
can be used to restrict the PC values for libffi trampolines to prevent hacking.
To generalize, we can implement security rules/features if the trampoline
object is in the kernel.
I don't follow this argument. If it's possible to statically define that
in the kernel, it's also possible to do that in userspace without any
new kernel support.
[...]
Trampfd is a framework that can be used to implement multiple things. May be,
a few of those things can also be implemented in user land itself. But I think having
just one mechanism to execute dynamic code objects is preferable to having
multiple mechanisms not standardized across all applications.
In abstract, having a common interface sounds nice, but in practice
elements of this are always architecture-specific (e.g. interactiosn
with HW CFI), and that common interface can result in more pain as it
doesn't fit naturally into the context that ISAs were designed for (e.g.
where control-flow instructions are extended with new semantics).
It also meass that you can't share the rough approach across OSs which
do not implement an identical mechanism, so for code abstracting by ISA
first, then by platform/ABI, there isn't much saving.
Thanks,
Mark.
From: David Laight <hidden> Date: 2020-08-03 08:27:33
From: Mark Rutland
Sent: 31 July 2020 19:32
...
quoted
It requires PC-relative data references. I have not worked on all architectures.
So, I need to study this. But do all ISAs support PC-relative data references?
Not all do, but pretty much any recent ISA will as it's a practical
necessity for fast position-independent code.
i386 has neither PC-relative addressing nor moves from %pc.
The cpu architecture knows that the sequence:
call 1f
1: pop %reg
is used to get the %pc value so is treated specially so that
it doesn't 'trash' the return stack.
So PIC code isn't too bad, but you have to use the correct
sequence.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
From: Madhavan T. Venkataraman <hidden> Date: 2020-08-03 16:03:25
On 8/3/20 3:27 AM, David Laight wrote:
From: Mark Rutland
quoted
Sent: 31 July 2020 19:32
...
quoted
quoted
It requires PC-relative data references. I have not worked on all architectures.
So, I need to study this. But do all ISAs support PC-relative data references?
Not all do, but pretty much any recent ISA will as it's a practical
necessity for fast position-independent code.
i386 has neither PC-relative addressing nor moves from %pc.
The cpu architecture knows that the sequence:
call 1f
1: pop %reg
is used to get the %pc value so is treated specially so that
it doesn't 'trash' the return stack.
So PIC code isn't too bad, but you have to use the correct
sequence.
Is that true only for 32-bit systems only? I thought RIP-relative addressing was
introduced in 64-bit mode. Please confirm.
Madhavan
From: David Laight <hidden> Date: 2020-08-03 16:57:54
From: Madhavan T. Venkataraman
Sent: 03 August 2020 17:03
On 8/3/20 3:27 AM, David Laight wrote:
quoted
From: Mark Rutland
quoted
Sent: 31 July 2020 19:32
...
quoted
quoted
It requires PC-relative data references. I have not worked on all architectures.
So, I need to study this. But do all ISAs support PC-relative data references?
Not all do, but pretty much any recent ISA will as it's a practical
necessity for fast position-independent code.
i386 has neither PC-relative addressing nor moves from %pc.
The cpu architecture knows that the sequence:
call 1f
1: pop %reg
is used to get the %pc value so is treated specially so that
it doesn't 'trash' the return stack.
So PIC code isn't too bad, but you have to use the correct
sequence.
Is that true only for 32-bit systems only? I thought RIP-relative addressing was
introduced in 64-bit mode. Please confirm.
I said i386 not amd64 or x86-64.
So yes, 64bit code has PC-relative addressing.
But I'm pretty sure it has no other way to get the PC itself
except using call - certainly nothing in the 'usual' instructions.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
From: Madhavan T. Venkataraman <hidden> Date: 2020-08-03 17:00:09
On 8/3/20 11:57 AM, David Laight wrote:
From: Madhavan T. Venkataraman
quoted
Sent: 03 August 2020 17:03
On 8/3/20 3:27 AM, David Laight wrote:
quoted
From: Mark Rutland
quoted
Sent: 31 July 2020 19:32
...
quoted
quoted
It requires PC-relative data references. I have not worked on all architectures.
So, I need to study this. But do all ISAs support PC-relative data references?
Not all do, but pretty much any recent ISA will as it's a practical
necessity for fast position-independent code.
i386 has neither PC-relative addressing nor moves from %pc.
The cpu architecture knows that the sequence:
call 1f
1: pop %reg
is used to get the %pc value so is treated specially so that
it doesn't 'trash' the return stack.
So PIC code isn't too bad, but you have to use the correct
sequence.
Is that true only for 32-bit systems only? I thought RIP-relative addressing was
introduced in 64-bit mode. Please confirm.
I said i386 not amd64 or x86-64.
I am sorry. My bad.
So yes, 64bit code has PC-relative addressing.
But I'm pretty sure it has no other way to get the PC itself
except using call - certainly nothing in the 'usual' instructions.
From: Madhavan T. Venkataraman <hidden> Date: 2020-08-03 17:58:11
On 7/31/20 1:31 PM, Mark Rutland wrote:
On Fri, Jul 31, 2020 at 12:13:49PM -0500, Madhavan T. Venkataraman wrote:
quoted
On 7/30/20 3:54 PM, Andy Lutomirski wrote:
quoted
On Thu, Jul 30, 2020 at 7:24 AM Madhavan T. Venkataraman
[off-list ref] wrote:
Dealing with multiple architectures
-----------------------------------------------
One good reason to use trampfd is multiple architecture support. The
trampoline table in a code page approach is neat. I don't deny that at
all. But my question is - can it be used in all cases?
It requires PC-relative data references. I have not worked on all architectures.
So, I need to study this. But do all ISAs support PC-relative data references?
Not all do, but pretty much any recent ISA will as it's a practical
necessity for fast position-independent code.
So, two questions:
1. IIUC, for position independent code, we need PC-relative control transfers. I know that
PC-relative control transfers are kinda fundamental. So, I expect most architectures
support it. But to implement the trampoline table suggestion, we need PC-relative
data references. Like:
movq X(%rip), %rax
2. Do you know which architectures do not support PC-relative data references? I am
going to study this. But if you have some information, I would appreciate it.
In any case, I think we should support all of the architectures on which Linux currently
runs even if they are legacy.
quoted
Even in an ISA that supports it, there would be a maximum supported offset
from the current PC that can be reached for a data reference. That maximum
needs to be at least the size of a base page in the architecture. This is because
the code page and the data page need to be separate for security reasons.
Do all ISAs support a sufficiently large offset?
ISAs with pc-relative addessing can usually generate PC-relative
addresses into a GPR, from which they can apply an arbitrarily large
offset.
I will study this. I need to nail down the list of architectures that cannot do this.
quoted
When the kernel generates the code for a trampoline, it can hard code data values
in the generated code itself so it does not need PC-relative data referencing.
And, for ISAs that do support the large offset, we do have to implement and
maintain the code page stuff for different ISAs for each application and library
if we did not use trampfd.
Trampoline code is architecture specific today, so I don't see that as a
major issue. Common structural bits can probably be shared even if the
specifid machine code cannot.
True. But an implementor may prefer a standard mechanism provided by
the kernel so all of his architectures can be supported easily with less
effort.
If you look at the libffi reference patch I have included, the architecture
specific changes to use trampfd just involve a single C function call to
a common code function.
So, from the point of view of adoption, IMHO, the kernel provided method
is preferable.
[...]
quoted
Security
-----------
With the user level trampoline table approach, the data part of the trampoline table
can be hacked by an attacker if an application has a vulnerability. Specifically, the
target PC can be altered to some arbitrary location. Trampfd implements an
"Allowed PCS" context. In the libffi changes, I have created a read-only array of
all ABI handlers used in closures for each architecture. This read-only array
can be used to restrict the PC values for libffi trampolines to prevent hacking.
To generalize, we can implement security rules/features if the trampoline
object is in the kernel.
I don't follow this argument. If it's possible to statically define that
in the kernel, it's also possible to do that in userspace without any
new kernel support.
It is not statically defined in the kernel.
Let us take the libffi example. In the 64-bit X86 arch code, there are 3
ABI handlers:
ffi_closure_unix64_sse
ffi_closure_unix64
ffi_closure_win64
I could create an "Allowed PCs" context like this:
struct my_allowed_pcs {
struct trampfd_values pcs;
__u64 pc_values[3];
};
const struct my_allowed_pcs my_allowed_pcs = {
{ 3, 0 },
(uintptr_t) ffi_closure_unix64_sse,
(uintptr_t) ffi_closure_unix64,
(uintptr_t) ffi_closure_win64,
};
I have created a read-only array of allowed ABI handlers that closures use.
When I set up the context for a closure trampoline, I could do this:
pwrite(trampfd, &my_allowed_pcs, sizeof(my_allowed_pcs), TRAMPFD_ALLOWED_PCS_OFFSET);
This copies the array into the trampoline object in the kernel.
When the register context is set for the trampoline, the kernel checks
the PC register value against allowed PCs.
Because my_allowed_pcs is read-only, a hacker cannot modify it. So, the only
permitted target PCs enforced by the kernel are the ABI handlers.
[...]
quoted
Trampfd is a framework that can be used to implement multiple things. May be,
a few of those things can also be implemented in user land itself. But I think having
just one mechanism to execute dynamic code objects is preferable to having
multiple mechanisms not standardized across all applications.
In abstract, having a common interface sounds nice, but in practice
elements of this are always architecture-specific (e.g. interactiosn
with HW CFI), and that common interface can result in more pain as it
doesn't fit naturally into the context that ISAs were designed for (e.g.
where control-flow instructions are extended with new semantics).
In the case of trampfd, the code generation is indeed architecture
specific. But that is in the kernel. The application is not affected by it.
Again, referring to the libffi reference patch, I have defined wrapper
functions for trampfd in common code. The architecture specific code
in libffi only calls the set_context function defined in common code.
Even this is required only because register names are specific to each
architecture and the target PC (to the ABI handler) is specific to
each architecture-ABI combo.
It also meass that you can't share the rough approach across OSs which
do not implement an identical mechanism, so for code abstracting by ISA
first, then by platform/ABI, there isn't much saving.
Why can you not share the same approach across OSes? In fact,
I have tried to design it so that other OSes can use the same
mechanism.
The only thing is that I have defined the API to be based on a file
descriptor since that is what is generally preferred by the Linux community
for a new API. If I were to implement it as a regular system call, the same
system call can be implemented in other OSes as well.
Thanks.
Madhavan
From: Mark Rutland <mark.rutland@arm.com> Date: 2020-08-04 13:56:16
On Mon, Aug 03, 2020 at 12:58:04PM -0500, Madhavan T. Venkataraman wrote:
On 7/31/20 1:31 PM, Mark Rutland wrote:
quoted
On Fri, Jul 31, 2020 at 12:13:49PM -0500, Madhavan T. Venkataraman wrote:
quoted
On 7/30/20 3:54 PM, Andy Lutomirski wrote:
quoted
On Thu, Jul 30, 2020 at 7:24 AM Madhavan T. Venkataraman
[off-list ref] wrote:
When the kernel generates the code for a trampoline, it can hard code data values
in the generated code itself so it does not need PC-relative data referencing.
And, for ISAs that do support the large offset, we do have to implement and
maintain the code page stuff for different ISAs for each application and library
if we did not use trampfd.
Trampoline code is architecture specific today, so I don't see that as a
major issue. Common structural bits can probably be shared even if the
specifid machine code cannot.
True. But an implementor may prefer a standard mechanism provided by
the kernel so all of his architectures can be supported easily with less
effort.
If you look at the libffi reference patch I have included, the architecture
specific changes to use trampfd just involve a single C function call to
a common code function.
Sure but in addition to that each architecture backend had to define a
set of arguments to that. I view the C function is analagous to the
"common structural bits".
I appreciate that your patch is small today (and architectures seem to
largely align on what they need), but I don't think it's necessarily
true that things will remain so simple as architecture are extended and
their calling conventions evolve, and I also don't think it's clear that
this will work for more complex cases elsewhere.
[...]
quoted
quoted
With the user level trampoline table approach, the data part of the trampoline table
can be hacked by an attacker if an application has a vulnerability. Specifically, the
target PC can be altered to some arbitrary location. Trampfd implements an
"Allowed PCS" context. In the libffi changes, I have created a read-only array of
all ABI handlers used in closures for each architecture. This read-only array
can be used to restrict the PC values for libffi trampolines to prevent hacking.
To generalize, we can implement security rules/features if the trampoline
object is in the kernel.
I don't follow this argument. If it's possible to statically define that
in the kernel, it's also possible to do that in userspace without any
new kernel support.
It is not statically defined in the kernel.
Let us take the libffi example. In the 64-bit X86 arch code, there are 3
ABI handlers:
ffi_closure_unix64_sse
ffi_closure_unix64
ffi_closure_win64
I could create an "Allowed PCs" context like this:
struct my_allowed_pcs {
struct trampfd_values pcs;
__u64 pc_values[3];
};
const struct my_allowed_pcs my_allowed_pcs = {
{ 3, 0 },
(uintptr_t) ffi_closure_unix64_sse,
(uintptr_t) ffi_closure_unix64,
(uintptr_t) ffi_closure_win64,
};
I have created a read-only array of allowed ABI handlers that closures use.
When I set up the context for a closure trampoline, I could do this:
pwrite(trampfd, &my_allowed_pcs, sizeof(my_allowed_pcs), TRAMPFD_ALLOWED_PCS_OFFSET);
This copies the array into the trampoline object in the kernel.
When the register context is set for the trampoline, the kernel checks
the PC register value against allowed PCs.
Because my_allowed_pcs is read-only, a hacker cannot modify it. So, the only
permitted target PCs enforced by the kernel are the ABI handlers.
Sorry, when I said "statically define" meant when you knew legitimate
targets ahead of time when you create the trampoline (i.e. whether you
could enumerate those and know they would not change dynamically).
My point was that you can achieve the same in userspace if the
trampoline and array of legitimate targets are in read-only memory,
without having to trap to the kernel.
I think the key point here is that an adversary must be prevented from
altering a trampoline and any associated metadata, and I think that
there are ways of achieving that without having to trap into the kernel,
and without the kernel having to be intimately aware of the calling
conventions used in userspace.
[...]
quoted
quoted
Trampfd is a framework that can be used to implement multiple things. May be,
a few of those things can also be implemented in user land itself. But I think having
just one mechanism to execute dynamic code objects is preferable to having
multiple mechanisms not standardized across all applications.
In abstract, having a common interface sounds nice, but in practice
elements of this are always architecture-specific (e.g. interactiosn
with HW CFI), and that common interface can result in more pain as it
doesn't fit naturally into the context that ISAs were designed for (e.g.
where control-flow instructions are extended with new semantics).
In the case of trampfd, the code generation is indeed architecture
specific. But that is in the kernel. The application is not affected by it.
As an ABI detail, applications are *definitely* affected by this, and it
is wrong to suggest they are not even if you don't have a specific case
in mind today. As this forms a contract between userspace and the kernel
it's overly simplistic to say that it's the kernel's problem
For example, in the case of BTI on arm64, what should the trampoline
set PSTATE.BTYPE to? Different use-cases *will* want different values,
and not necessarily the value of PSTATE at the instant the call to the
trampoline was made. In the case of libffi specifically using the
original value of PSTATE.BTYPE probably is sound, but other code
sequences may need to restrict/broaden or entirely change that.
Again, referring to the libffi reference patch, I have defined wrapper
functions for trampfd in common code. The architecture specific code
in libffi only calls the set_context function defined in common code.
Even this is required only because register names are specific to each
architecture and the target PC (to the ABI handler) is specific to
each architecture-ABI combo.
quoted
It also meass that you can't share the rough approach across OSs which
do not implement an identical mechanism, so for code abstracting by ISA
first, then by platform/ABI, there isn't much saving.
Why can you not share the same approach across OSes? In fact,
I have tried to design it so that other OSes can use the same
mechanism.
Sure, but where they *don't*, you must fall back to the existing
purely-userspace mechanisms, and so a codebase now has the burden of
maintaining two distinct mechanisms.
Whereas if there's a way of doing this in userspace with (stronger)
enforcement of memory permissions the trampoline code can be common for
when this is present or absent, which is much easier for a codebase rto
maintain, and could make use of weaker existing mechanisms to improve
the situation on systems without the new functionality.
Thanks,
Mark.
From: David Laight <hidden> Date: 2020-08-04 14:39:57
quoted
If you look at the libffi reference patch I have included, the architecture
specific changes to use trampfd just involve a single C function call to
a common code function.
No idea what libffi is, but it must surely be simpler to
rewrite it to avoid nested function definitions.
Or find a book from the 1960s on how to do recursive
calls and nested functions in FORTRAN-IV.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
From: David Laight <hidden> Date: 2020-08-04 14:45:08
quoted
quoted
If you look at the libffi reference patch I have included, the architecture
specific changes to use trampfd just involve a single C function call to
a common code function.
No idea what libffi is, but it must surely be simpler to
rewrite it to avoid nested function definitions.
Or find a book from the 1960s on how to do recursive
calls and nested functions in FORTRAN-IV.
FWIW it is probably as simple as:
1) Put all the 'variables' the nested function accesses into a struct.
2) Add a field for the address of the 'nested' function.
3) Pass the address of the structure down instead of the
address of the function.
If you aren't in control of the call sites then add the
structure to a linked list on a thread-local variable.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
From: Madhavan T. Venkataraman <hidden> Date: 2020-08-04 14:48:20
On 8/4/20 9:33 AM, David Laight wrote:
quoted
quoted
If you look at the libffi reference patch I have included, the architecture
specific changes to use trampfd just involve a single C function call to
a common code function.
No idea what libffi is, but it must surely be simpler to
rewrite it to avoid nested function definitions.
Sorry if I wasn't clear.
libffi is a separate use case and GCC nested functions is a separate one.
libffi is not used to solve the nested function stuff.
For nested functions, GCC generates trampoline code and arranges to
place it on the stack and execute it.
I agree with your other points about nested function implementation.
Madhavan
Or find a book from the 1960s on how to do recursive
calls and nested functions in FORTRAN-IV.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
From: Madhavan T. Venkataraman <hidden> Date: 2020-08-04 15:46:54
Hey Mark,
I am working on putting together an improved definition of trampfd per
Andy's comment. I will try to address your comments in that improved
definition. Once I send that out, I will respond to your emails as well.
Thanks.
Madhavan
On 8/4/20 8:55 AM, Mark Rutland wrote:
On Mon, Aug 03, 2020 at 12:58:04PM -0500, Madhavan T. Venkataraman wrote:
quoted
On 7/31/20 1:31 PM, Mark Rutland wrote:
quoted
On Fri, Jul 31, 2020 at 12:13:49PM -0500, Madhavan T. Venkataraman wrote:
quoted
On 7/30/20 3:54 PM, Andy Lutomirski wrote:
quoted
On Thu, Jul 30, 2020 at 7:24 AM Madhavan T. Venkataraman
[off-list ref] wrote:
When the kernel generates the code for a trampoline, it can hard code data values
in the generated code itself so it does not need PC-relative data referencing.
And, for ISAs that do support the large offset, we do have to implement and
maintain the code page stuff for different ISAs for each application and library
if we did not use trampfd.
Trampoline code is architecture specific today, so I don't see that as a
major issue. Common structural bits can probably be shared even if the
specifid machine code cannot.
True. But an implementor may prefer a standard mechanism provided by
the kernel so all of his architectures can be supported easily with less
effort.
If you look at the libffi reference patch I have included, the architecture
specific changes to use trampfd just involve a single C function call to
a common code function.
Sure but in addition to that each architecture backend had to define a
set of arguments to that. I view the C function is analagous to the
"common structural bits".
I appreciate that your patch is small today (and architectures seem to
largely align on what they need), but I don't think it's necessarily
true that things will remain so simple as architecture are extended and
their calling conventions evolve, and I also don't think it's clear that
this will work for more complex cases elsewhere.
[...]
quoted
quoted
quoted
With the user level trampoline table approach, the data part of the trampoline table
can be hacked by an attacker if an application has a vulnerability. Specifically, the
target PC can be altered to some arbitrary location. Trampfd implements an
"Allowed PCS" context. In the libffi changes, I have created a read-only array of
all ABI handlers used in closures for each architecture. This read-only array
can be used to restrict the PC values for libffi trampolines to prevent hacking.
To generalize, we can implement security rules/features if the trampoline
object is in the kernel.
I don't follow this argument. If it's possible to statically define that
in the kernel, it's also possible to do that in userspace without any
new kernel support.
It is not statically defined in the kernel.
Let us take the libffi example. In the 64-bit X86 arch code, there are 3
ABI handlers:
ffi_closure_unix64_sse
ffi_closure_unix64
ffi_closure_win64
I could create an "Allowed PCs" context like this:
struct my_allowed_pcs {
struct trampfd_values pcs;
__u64 pc_values[3];
};
const struct my_allowed_pcs my_allowed_pcs = {
{ 3, 0 },
(uintptr_t) ffi_closure_unix64_sse,
(uintptr_t) ffi_closure_unix64,
(uintptr_t) ffi_closure_win64,
};
I have created a read-only array of allowed ABI handlers that closures use.
When I set up the context for a closure trampoline, I could do this:
pwrite(trampfd, &my_allowed_pcs, sizeof(my_allowed_pcs), TRAMPFD_ALLOWED_PCS_OFFSET);
This copies the array into the trampoline object in the kernel.
When the register context is set for the trampoline, the kernel checks
the PC register value against allowed PCs.
Because my_allowed_pcs is read-only, a hacker cannot modify it. So, the only
permitted target PCs enforced by the kernel are the ABI handlers.
Sorry, when I said "statically define" meant when you knew legitimate
targets ahead of time when you create the trampoline (i.e. whether you
could enumerate those and know they would not change dynamically).
My point was that you can achieve the same in userspace if the
trampoline and array of legitimate targets are in read-only memory,
without having to trap to the kernel.
I think the key point here is that an adversary must be prevented from
altering a trampoline and any associated metadata, and I think that
there are ways of achieving that without having to trap into the kernel,
and without the kernel having to be intimately aware of the calling
conventions used in userspace.
[...]
quoted
quoted
quoted
Trampfd is a framework that can be used to implement multiple things. May be,
a few of those things can also be implemented in user land itself. But I think having
just one mechanism to execute dynamic code objects is preferable to having
multiple mechanisms not standardized across all applications.
In abstract, having a common interface sounds nice, but in practice
elements of this are always architecture-specific (e.g. interactiosn
with HW CFI), and that common interface can result in more pain as it
doesn't fit naturally into the context that ISAs were designed for (e.g.
where control-flow instructions are extended with new semantics).
In the case of trampfd, the code generation is indeed architecture
specific. But that is in the kernel. The application is not affected by it.
As an ABI detail, applications are *definitely* affected by this, and it
is wrong to suggest they are not even if you don't have a specific case
in mind today. As this forms a contract between userspace and the kernel
it's overly simplistic to say that it's the kernel's problem
For example, in the case of BTI on arm64, what should the trampoline
set PSTATE.BTYPE to? Different use-cases *will* want different values,
and not necessarily the value of PSTATE at the instant the call to the
trampoline was made. In the case of libffi specifically using the
original value of PSTATE.BTYPE probably is sound, but other code
sequences may need to restrict/broaden or entirely change that.
quoted
Again, referring to the libffi reference patch, I have defined wrapper
functions for trampfd in common code. The architecture specific code
in libffi only calls the set_context function defined in common code.
Even this is required only because register names are specific to each
architecture and the target PC (to the ABI handler) is specific to
each architecture-ABI combo.
quoted
It also meass that you can't share the rough approach across OSs which
do not implement an identical mechanism, so for code abstracting by ISA
first, then by platform/ABI, there isn't much saving.
Why can you not share the same approach across OSes? In fact,
I have tried to design it so that other OSes can use the same
mechanism.
Sure, but where they *don't*, you must fall back to the existing
purely-userspace mechanisms, and so a codebase now has the burden of
maintaining two distinct mechanisms.
Whereas if there's a way of doing this in userspace with (stronger)
enforcement of memory permissions the trampoline code can be common for
when this is present or absent, which is much easier for a codebase rto
maintain, and could make use of weaker existing mechanisms to improve
the situation on systems without the new functionality.
Thanks,
Mark.