On Wed, Jan 13, 2021 at 04:57:43PM +0000, Mark Brown wrote:
From: Mark Rutland <mark.rutland@arm.com>
Add documentation for reliable stacktrace. This is intended to describe
the semantics and to be an aid for implementing architecture support for
HAVE_RELIABLE_STACKTRACE.
Unwinding is a subtle area, and architectures vary greatly in both
implementation and the set of concerns that affect them, so I've tried
to avoid making this too specific to any given architecture. I've used
examples from both x86_64 and arm64 to explain corner cases in more
detail, but I've tried to keep the descriptions sufficient for those who
are unfamiliar with the particular architecture.
I've tried to give rationale for all the recommendations/requirements,
since that makes it easier to spot nearby issues, or when a check
happens to catch a few things at once. I believe what I have written is
sound, but as some of this was reverse-engineered I may have missed
things worth noting.
I've made a few assumptions about preferred behaviour, notably:
* If you can reliably unwind through exceptions, you should (as x86_64
does).
* It's fine to omit ftrace_return_to_handler and other return
trampolines so long as these are not subject to patching and the
original return address is reported. Most architectures do this for
ftrace_return_handler, but not other return trampolines.
* For cases where link register unreliability could result in duplicate
entries in the trace or an inverted trace, I've assumed this should be
treated as unreliable. This specific case shouldn't matter to
livepatching, but I assume that that we want a reliable trace to have
the correct order.
Thanks to you and Mark for getting this documented properly!
I think it's worth mentioning a little more about objtool. There are a
few passing mentions of objtool's generation of metadata (i.e. ORC), but
objtool has another relevant purpose: stack validation. That's
particularly important when it comes to frame pointers.
For some architectures like x86_64 and arm64 (but not powerpc/s390),
it's far too easy for a human to write asm and/or inline asm which
violates frame pointer protocol, silently causing the violater's callee
to get skipped in the unwind. Such architectures need objtool
implemented for CONFIG_STACK_VALIDATION.
+There are several ways an architecture may identify kernel code which is deemed
+unreliable to unwind from, e.g.
+
+* Using metadata created by objtool, with such code annotated with
+ SYM_CODE_{START,END} or STACKFRAME_NON_STANDARD().
I'm not sure why SYM_CODE_{START,END} is mentioned here, but it doesn't
necessarily mean the code is unreliable, and objtool doesn't treat it as
such. Its mention can probably be removed unless there was some other
point I'm missing.
Also, s/STACKFRAME/STACK_FRAME/
--
Josh
From: Mark Brown <broonie@kernel.org> Date: 2021-01-13 20:24:48
On Wed, Jan 13, 2021 at 01:33:13PM -0600, Josh Poimboeuf wrote:
I think it's worth mentioning a little more about objtool. There are a
few passing mentions of objtool's generation of metadata (i.e. ORC), but
objtool has another relevant purpose: stack validation. That's
particularly important when it comes to frame pointers.
For some architectures like x86_64 and arm64 (but not powerpc/s390),
it's far too easy for a human to write asm and/or inline asm which
violates frame pointer protocol, silently causing the violater's callee
to get skipped in the unwind. Such architectures need objtool
implemented for CONFIG_STACK_VALIDATION.
This basically boils down to just adding a statement saying "you may
need to depend on objtool" I think?
quoted
+There are several ways an architecture may identify kernel code which is deemed
+unreliable to unwind from, e.g.
quoted
+* Using metadata created by objtool, with such code annotated with
+ SYM_CODE_{START,END} or STACKFRAME_NON_STANDARD().
I'm not sure why SYM_CODE_{START,END} is mentioned here, but it doesn't
necessarily mean the code is unreliable, and objtool doesn't treat it as
such. Its mention can probably be removed unless there was some other
point I'm missing.
I was reading that as being a thing that the architecture could possibly
do, especially as a first step - it does seem like a reasonable thing to
consider using anyway. I guess you could also use it the other way
around and do additional checks for things that are supposed to be
regular functions that you relax for SYM_CODE() sections.
On Wed, Jan 13, 2021 at 08:23:15PM +0000, Mark Brown wrote:
On Wed, Jan 13, 2021 at 01:33:13PM -0600, Josh Poimboeuf wrote:
quoted
I think it's worth mentioning a little more about objtool. There are a
few passing mentions of objtool's generation of metadata (i.e. ORC), but
objtool has another relevant purpose: stack validation. That's
particularly important when it comes to frame pointers.
quoted
For some architectures like x86_64 and arm64 (but not powerpc/s390),
it's far too easy for a human to write asm and/or inline asm which
violates frame pointer protocol, silently causing the violater's callee
to get skipped in the unwind. Such architectures need objtool
implemented for CONFIG_STACK_VALIDATION.
This basically boils down to just adding a statement saying "you may
need to depend on objtool" I think?
Right, but maybe it would be a short paragraph or two.
quoted
quoted
+There are several ways an architecture may identify kernel code which is deemed
+unreliable to unwind from, e.g.
quoted
quoted
+* Using metadata created by objtool, with such code annotated with
+ SYM_CODE_{START,END} or STACKFRAME_NON_STANDARD().
quoted
I'm not sure why SYM_CODE_{START,END} is mentioned here, but it doesn't
necessarily mean the code is unreliable, and objtool doesn't treat it as
such. Its mention can probably be removed unless there was some other
point I'm missing.
I was reading that as being a thing that the architecture could possibly
do, especially as a first step - it does seem like a reasonable thing to
consider using anyway. I guess you could also use it the other way
around and do additional checks for things that are supposed to be
regular functions that you relax for SYM_CODE() sections.
Makes sense, but we have to be careful not to imply that objtool already
does something like that :-)
--
Josh
From: Mark Rutland <mark.rutland@arm.com> Date: 2021-01-14 18:11:22
On Wed, Jan 13, 2021 at 04:25:41PM -0600, Josh Poimboeuf wrote:
On Wed, Jan 13, 2021 at 08:23:15PM +0000, Mark Brown wrote:
quoted
On Wed, Jan 13, 2021 at 01:33:13PM -0600, Josh Poimboeuf wrote:
quoted
I think it's worth mentioning a little more about objtool. There are a
few passing mentions of objtool's generation of metadata (i.e. ORC), but
objtool has another relevant purpose: stack validation. That's
particularly important when it comes to frame pointers.
quoted
For some architectures like x86_64 and arm64 (but not powerpc/s390),
it's far too easy for a human to write asm and/or inline asm which
violates frame pointer protocol, silently causing the violater's callee
to get skipped in the unwind. Such architectures need objtool
implemented for CONFIG_STACK_VALIDATION.
This basically boils down to just adding a statement saying "you may
need to depend on objtool" I think?
Right, but maybe it would be a short paragraph or two.
I reckon that's a top-level section between requirements and
consideration along the lines of:
3. Compile-time analysis
========================
To ensure that kernel code can be correctly unwound in all cases,
architectures may need to verify that code has been compiled in a manner
expected by the unwinder. For example, an unwinder may expect that
functions manipulate the stack pointer in a limited way, or that all
functions use specific prologue and epilogue sequences. Architectures
with such requirements should verify the kernel compilation using
objtool.
In some cases, an unwinder may require metadata to correctly unwind.
Where necessary, this metadata should be generated at build time using
objtool.
... perhaps elaborating a little further on the latter?
Thanks,
Mark.
On Thu, Jan 14, 2021 at 06:10:13PM +0000, Mark Rutland wrote:
On Wed, Jan 13, 2021 at 04:25:41PM -0600, Josh Poimboeuf wrote:
quoted
On Wed, Jan 13, 2021 at 08:23:15PM +0000, Mark Brown wrote:
quoted
On Wed, Jan 13, 2021 at 01:33:13PM -0600, Josh Poimboeuf wrote:
quoted
I think it's worth mentioning a little more about objtool. There are a
few passing mentions of objtool's generation of metadata (i.e. ORC), but
objtool has another relevant purpose: stack validation. That's
particularly important when it comes to frame pointers.
quoted
For some architectures like x86_64 and arm64 (but not powerpc/s390),
it's far too easy for a human to write asm and/or inline asm which
violates frame pointer protocol, silently causing the violater's callee
to get skipped in the unwind. Such architectures need objtool
implemented for CONFIG_STACK_VALIDATION.
This basically boils down to just adding a statement saying "you may
need to depend on objtool" I think?
Right, but maybe it would be a short paragraph or two.
I reckon that's a top-level section between requirements and
consideration along the lines of:
3. Compile-time analysis
========================
To ensure that kernel code can be correctly unwound in all cases,
architectures may need to verify that code has been compiled in a manner
expected by the unwinder. For example, an unwinder may expect that
functions manipulate the stack pointer in a limited way, or that all
functions use specific prologue and epilogue sequences. Architectures
with such requirements should verify the kernel compilation using
objtool.
In some cases, an unwinder may require metadata to correctly unwind.
Where necessary, this metadata should be generated at build time using
objtool.
From: Mark Rutland <mark.rutland@arm.com> Date: 2021-01-14 11:55:26
On Wed, Jan 13, 2021 at 01:33:13PM -0600, Josh Poimboeuf wrote:
On Wed, Jan 13, 2021 at 04:57:43PM +0000, Mark Brown wrote:
quoted
From: Mark Rutland <mark.rutland@arm.com>
+There are several ways an architecture may identify kernel code which is deemed
+unreliable to unwind from, e.g.
+
+* Using metadata created by objtool, with such code annotated with
+ SYM_CODE_{START,END} or STACKFRAME_NON_STANDARD().
I'm not sure why SYM_CODE_{START,END} is mentioned here, but it doesn't
necessarily mean the code is unreliable, and objtool doesn't treat it as
such. Its mention can probably be removed unless there was some other
point I'm missing.
Also, s/STACKFRAME/STACK_FRAME/
When I wrote this, I was under the impression that (for x86) code marked
as SYM_CODE_{START,END} wouldn't be considered as a function by objtool.
Specifically SYM_FUNC_END() marks the function with SYM_T_FUNC whereas
SYM_CODE_END() marks it with SYM_T_NONE, and IIRC I thought that objtool
only generated ORC for SYM_T_FUNC functions, and hence anything else
would be considered not unwindable due to the absence of ORC.
Just to check, is that understanding for x86 correct, or did I get that
wrong?
If that's right, it might be worth splitting this into two points, e.g.
| * Using metadata created by objtool, with such code annotated with
| STACKFRAME_NON_STANDARD().
|
|
| * Using ELF symbol attributes, with such code annotated with
| SYM_CODE_{START,END}, and not having a function type.
If that's wrong, I suspect there are latent issues here?
Thanks,
Mark.
On Thu, Jan 14, 2021 at 11:54:18AM +0000, Mark Rutland wrote:
On Wed, Jan 13, 2021 at 01:33:13PM -0600, Josh Poimboeuf wrote:
quoted
On Wed, Jan 13, 2021 at 04:57:43PM +0000, Mark Brown wrote:
quoted
From: Mark Rutland <mark.rutland@arm.com>
+There are several ways an architecture may identify kernel code which is deemed
+unreliable to unwind from, e.g.
+
+* Using metadata created by objtool, with such code annotated with
+ SYM_CODE_{START,END} or STACKFRAME_NON_STANDARD().
I'm not sure why SYM_CODE_{START,END} is mentioned here, but it doesn't
necessarily mean the code is unreliable, and objtool doesn't treat it as
such. Its mention can probably be removed unless there was some other
point I'm missing.
Also, s/STACKFRAME/STACK_FRAME/
When I wrote this, I was under the impression that (for x86) code marked
as SYM_CODE_{START,END} wouldn't be considered as a function by objtool.
Specifically SYM_FUNC_END() marks the function with SYM_T_FUNC whereas
SYM_CODE_END() marks it with SYM_T_NONE, and IIRC I thought that objtool
only generated ORC for SYM_T_FUNC functions, and hence anything else
would be considered not unwindable due to the absence of ORC.
Just to check, is that understanding for x86 correct, or did I get that
wrong?
Doh, I suppose you read the documentation ;-)
I realize your understanding is pretty much consistent with
tools/objtool/Documentation/stack-validation.txt:
2. Conversely, each section of code which is *not* callable should *not*
be annotated as an ELF function. The ENDPROC macro shouldn't be used
in this case.
This rule is needed so that objtool can ignore non-callable code.
Such code doesn't have to follow any of the other rules.
But this statement is no longer true:
**This rule is needed so that objtool can ignore non-callable code.**
[ and it looks like the ENDPROC reference is also out of date ]
Since that document was written, around the time ORC was written we
realized objtool shouldn't ignore SYM_CODE after all. That way we can
get full coverage for ORC (including interrupts/exceptions), as well as
some of the other validations like retpoline, uaccess, noinstr, etc.
Though it's still true that SYM_CODE doesn't have to follow the
function-specific rules, e.g. frame pointers.
So now objtool requires that it be able to traverse and understand *all*
code, otherwise it will spit out "unreachable instruction" warnings.
But since SYM_CODE isn't a normal callable function, objtool doesn't
know to interpret it directly. Therefore all SYM_CODE must be reachable
by objtool in some other way:
- either indirectly, via a jump from a SYM_FUNC; or
- via an UNWIND_HINT
(And that's true for both ORC and frame pointers.)
If you look closely at arch/x86/entry/entry_64.S you should be able to
see that's the case.
If that's right, it might be worth splitting this into two points, e.g.
| * Using metadata created by objtool, with such code annotated with
| STACKFRAME_NON_STANDARD().
|
|
| * Using ELF symbol attributes, with such code annotated with
| SYM_CODE_{START,END}, and not having a function type.
If that's wrong, I suspect there are latent issues here?
For ORC, UNWIND_HINT_EMPTY is used to annotate that some code is
non-unwindable. (Note I have plans to split that into UNWIND_HINT_ENTRY
and UNWIND_HINT_UNDEFINED.)
For frame pointers, the hints aren't used, other than by objtool to
follow the code flow as described above. But objtool doesn't produce
any metadata for the FP unwinder. Instead the FP unwinder makes such
determinations about unwindability at runtime.
--
Josh
From: Mark Rutland <mark.rutland@arm.com> Date: 2021-01-14 17:50:46
On Thu, Jan 14, 2021 at 08:36:50AM -0600, Josh Poimboeuf wrote:
On Thu, Jan 14, 2021 at 11:54:18AM +0000, Mark Rutland wrote:
quoted
On Wed, Jan 13, 2021 at 01:33:13PM -0600, Josh Poimboeuf wrote:
quoted
On Wed, Jan 13, 2021 at 04:57:43PM +0000, Mark Brown wrote:
quoted
From: Mark Rutland <mark.rutland@arm.com>
+There are several ways an architecture may identify kernel code which is deemed
+unreliable to unwind from, e.g.
+
+* Using metadata created by objtool, with such code annotated with
+ SYM_CODE_{START,END} or STACKFRAME_NON_STANDARD().
I'm not sure why SYM_CODE_{START,END} is mentioned here, but it doesn't
necessarily mean the code is unreliable, and objtool doesn't treat it as
such. Its mention can probably be removed unless there was some other
point I'm missing.
quoted
Also, s/STACKFRAME/STACK_FRAME/
Given that (per the discussion below) STACK_FRAME_NON_STANDARD() also
doesn't result in objtool producing anything special metadata (and such
code is still expected to be unwindable), I believe we can delete this
bullet point outright?
quoted
When I wrote this, I was under the impression that (for x86) code marked
as SYM_CODE_{START,END} wouldn't be considered as a function by objtool.
Specifically SYM_FUNC_END() marks the function with SYM_T_FUNC whereas
SYM_CODE_END() marks it with SYM_T_NONE, and IIRC I thought that objtool
only generated ORC for SYM_T_FUNC functions, and hence anything else
would be considered not unwindable due to the absence of ORC.
Just to check, is that understanding for x86 correct, or did I get that
wrong?
Doh, I suppose you read the documentation ;-)
I think I skimmed the objtool source, too, but it was a while back. ;)
I realize your understanding is pretty much consistent with
tools/objtool/Documentation/stack-validation.txt:
2. Conversely, each section of code which is *not* callable should *not*
be annotated as an ELF function. The ENDPROC macro shouldn't be used
in this case.
This rule is needed so that objtool can ignore non-callable code.
Such code doesn't have to follow any of the other rules.
But this statement is no longer true:
**This rule is needed so that objtool can ignore non-callable code.**
[ and it looks like the ENDPROC reference is also out of date ]
Ok -- looks like that needs an update!
Since that document was written, around the time ORC was written we
realized objtool shouldn't ignore SYM_CODE after all. That way we can
get full coverage for ORC (including interrupts/exceptions), as well as
some of the other validations like retpoline, uaccess, noinstr, etc.
Though it's still true that SYM_CODE doesn't have to follow the
function-specific rules, e.g. frame pointers.
Ok; I suspect on the arm64 side we'll need to think a bit harder about
what that means for us. I guess that'll influence or interact with
whatever support we need specifically in objtool.
So now objtool requires that it be able to traverse and understand *all*
code, otherwise it will spit out "unreachable instruction" warnings.
But since SYM_CODE isn't a normal callable function, objtool doesn't
know to interpret it directly. Therefore all SYM_CODE must be reachable
by objtool in some other way:
- either indirectly, via a jump from a SYM_FUNC; or
- via an UNWIND_HINT
(And that's true for both ORC and frame pointers.)
If you look closely at arch/x86/entry/entry_64.S you should be able to
see that's the case.
Assuming you mean the UNWIND_HINT_EMPTY at the start of each exception
entry point, I think I follow.
quoted
If that's right, it might be worth splitting this into two points, e.g.
| * Using metadata created by objtool, with such code annotated with
| STACKFRAME_NON_STANDARD().
|
|
| * Using ELF symbol attributes, with such code annotated with
| SYM_CODE_{START,END}, and not having a function type.
If that's wrong, I suspect there are latent issues here?
For ORC, UNWIND_HINT_EMPTY is used to annotate that some code is
non-unwindable. (Note I have plans to split that into UNWIND_HINT_ENTRY
and UNWIND_HINT_UNDEFINED.)
Interesting; where would the UNDEFINED case be used?
For frame pointers, the hints aren't used, other than by objtool to
follow the code flow as described above. But objtool doesn't produce
any metadata for the FP unwinder. Instead the FP unwinder makes such
determinations about unwindability at runtime.
I suspect for arm64 with frame pointers we'll need a fair amount of
special casing for the entry code; I'll have a think offline.
Thanks,
Mark.
On Thu, Jan 14, 2021 at 05:49:32PM +0000, Mark Rutland wrote:
On Thu, Jan 14, 2021 at 08:36:50AM -0600, Josh Poimboeuf wrote:
quoted
On Thu, Jan 14, 2021 at 11:54:18AM +0000, Mark Rutland wrote:
quoted
On Wed, Jan 13, 2021 at 01:33:13PM -0600, Josh Poimboeuf wrote:
quoted
On Wed, Jan 13, 2021 at 04:57:43PM +0000, Mark Brown wrote:
quoted
From: Mark Rutland <mark.rutland@arm.com>
+There are several ways an architecture may identify kernel code which is deemed
+unreliable to unwind from, e.g.
+
+* Using metadata created by objtool, with such code annotated with
+ SYM_CODE_{START,END} or STACKFRAME_NON_STANDARD().
I'm not sure why SYM_CODE_{START,END} is mentioned here, but it doesn't
necessarily mean the code is unreliable, and objtool doesn't treat it as
such. Its mention can probably be removed unless there was some other
point I'm missing.
quoted
Also, s/STACKFRAME/STACK_FRAME/
Given that (per the discussion below) STACK_FRAME_NON_STANDARD() also
doesn't result in objtool producing anything special metadata (and such
code is still expected to be unwindable), I believe we can delete this
bullet point outright?
With ORC, UNWIND_HINT_EMPTY can be used to mark missing ORC metadata,
which the unwinder translates as unreliable. So that may be worth
mentioning.
quoted
I realize your understanding is pretty much consistent with
tools/objtool/Documentation/stack-validation.txt:
2. Conversely, each section of code which is *not* callable should *not*
be annotated as an ELF function. The ENDPROC macro shouldn't be used
in this case.
This rule is needed so that objtool can ignore non-callable code.
Such code doesn't have to follow any of the other rules.
But this statement is no longer true:
**This rule is needed so that objtool can ignore non-callable code.**
[ and it looks like the ENDPROC reference is also out of date ]
Ok -- looks like that needs an update!
Added to the TODO list!
quoted
Since that document was written, around the time ORC was written we
realized objtool shouldn't ignore SYM_CODE after all. That way we can
get full coverage for ORC (including interrupts/exceptions), as well as
some of the other validations like retpoline, uaccess, noinstr, etc.
Though it's still true that SYM_CODE doesn't have to follow the
function-specific rules, e.g. frame pointers.
Ok; I suspect on the arm64 side we'll need to think a bit harder about
what that means for us. I guess that'll influence or interact with
whatever support we need specifically in objtool.
quoted
So now objtool requires that it be able to traverse and understand *all*
code, otherwise it will spit out "unreachable instruction" warnings.
But since SYM_CODE isn't a normal callable function, objtool doesn't
know to interpret it directly. Therefore all SYM_CODE must be reachable
by objtool in some other way:
- either indirectly, via a jump from a SYM_FUNC; or
This should say "via a jump from some code objtool already knows about:
either a SYM_FUNC or a SYM_CODE with UNWIND_HINTs".
quoted
- via an UNWIND_HINT
(And that's true for both ORC and frame pointers.)
If you look closely at arch/x86/entry/entry_64.S you should be able to
see that's the case.
Assuming you mean the UNWIND_HINT_EMPTY at the start of each exception
entry point, I think I follow.
Also see for example common_interrupt_return(), which doesn't have an
UNWIND_HINT right away, but is still reachable from other code which
objtool already knows about via
the 'swapgs_restore_regs_and_return_to_usermode' label.
quoted
quoted
If that's right, it might be worth splitting this into two points, e.g.
| * Using metadata created by objtool, with such code annotated with
| STACKFRAME_NON_STANDARD().
|
|
| * Using ELF symbol attributes, with such code annotated with
| SYM_CODE_{START,END}, and not having a function type.
If that's wrong, I suspect there are latent issues here?
For ORC, UNWIND_HINT_EMPTY is used to annotate that some code is
non-unwindable. (Note I have plans to split that into UNWIND_HINT_ENTRY
and UNWIND_HINT_UNDEFINED.)
Interesting; where would the UNDEFINED case be used?
UNDEFINED would be some code which is either unreachable (like the
middle of a retpoline) or otherwise not annotated (like
STACK_FRAME_NON_STANDARD).
quoted
For frame pointers, the hints aren't used, other than by objtool to
follow the code flow as described above. But objtool doesn't produce
any metadata for the FP unwinder. Instead the FP unwinder makes such
determinations about unwindability at runtime.
I suspect for arm64 with frame pointers we'll need a fair amount of
special casing for the entry code; I'll have a think offline.
I'd be happy to help with this. It may end up easier for me to learn
your entry code than for you to learn the expectations of my tool ;-)
--
Josh