From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:16:33
From: Yu-cheng Yu <redacted>
Introduce a new document on Control-flow Enforcement Technology (CET).
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v5:
- Literal format tweaks (Bagas Sanjaya)
- Update EOPNOTSUPP text due to unification after comment from (Kees)
- Update 32 bit signal support with new behavior
- Remove capitalization on shadow stack (Boris)
- Fix typo
v4:
- Drop clearcpuid piece (Boris)
- Add some info about 32 bit
v3:
- Clarify kernel IBT is supported by the kernel. (Kees, Andrew Cooper)
- Clarify which arch_prctl's can take multiple bits. (Kees)
- Describe ASLR characteristics of thread shadow stacks. (Kees)
- Add exec section. (Andrew Cooper)
- Fix some capitalization (Bagas Sanjaya)
- Update new location of enablement status proc.
- Add info about new user_shstk software capability.
- Add more info about what the kernel pushes to the shadow stack on
signal.
v2:
- Updated to new arch_prctl() API
- Add bit about new proc status
---
Documentation/x86/index.rst | 1 +
Documentation/x86/shstk.rst | 166 ++++++++++++++++++++++++++++++++++++
2 files changed, 167 insertions(+)
create mode 100644 Documentation/x86/shstk.rst
@@ -0,0 +1,166 @@+.. SPDX-License-Identifier: GPL-2.0++======================================================+Control-flow Enforcement Technology (CET) Shadow Stack+======================================================++CET Background+==============++Control-flow Enforcement Technology (CET) is term referring to several+related x86 processor features that provides protection against control+flow hijacking attacks. The HW feature itself can be set up to protect+both applications and the kernel.++CET introduces shadow stack and indirect branch tracking (IBT). Shadow stack+is a secondary stack allocated from memory and cannot be directly modified by+applications. When executing a CALL instruction, the processor pushes the+return address to both the normal stack and the shadow stack. Upon+function return, the processor pops the shadow stack copy and compares it+to the normal stack copy. If the two differ, the processor raises a+control-protection fault. IBT verifies indirect CALL/JMP targets are intended+as marked by the compiler with 'ENDBR' opcodes. Not all CPU's have both Shadow+Stack and Indirect Branch Tracking. Today in the 64-bit kernel, only userspace+shadow stack and kernel IBT are supported.++Requirements to use Shadow Stack+================================++To use userspace shadow stack you need HW that supports it, a kernel+configured with it and userspace libraries compiled with it.++The kernel Kconfig option is X86_USER_SHADOW_STACK, and it can be disabled+with the kernel parameter: nousershstk.++To build a user shadow stack enabled kernel, Binutils v2.29 or LLVM v6 or later+are required.++At run time, /proc/cpuinfo shows CET features if the processor supports+CET. "user_shstk" means that userspace shadow stack is supported on the current+kernel and HW.++Application Enabling+====================++An application's CET capability is marked in its ELF note and can be verified+from readelf/llvm-readelf output::++ readelf -n <application> | grep -a SHSTK+ properties: x86 feature: SHSTK++The kernel does not process these applications markers directly. Applications+or loaders must enable CET features using the interface described in section 4.+Typically this would be done in dynamic loader or static runtime objects, as is+the case in GLIBC.++Enabling arch_prctl()'s+=======================++Elf features should be enabled by the loader using the below arch_prctl's. They+are only supported in 64 bit user applications.++arch_prctl(ARCH_SHSTK_ENABLE, unsigned long feature)+ Enable a single feature specified in 'feature'. Can only operate on+ one feature at a time.++arch_prctl(ARCH_SHSTK_DISABLE, unsigned long feature)+ Disable a single feature specified in 'feature'. Can only operate on+ one feature at a time.++arch_prctl(ARCH_SHSTK_LOCK, unsigned long features)+ Lock in features at their current enabled or disabled status. 'features'+ is a mask of all features to lock. All bits set are processed, unset bits+ are ignored. The mask is ORed with the existing value. So any feature bits+ set here cannot be enabled or disabled afterwards.++The return values are as follows. On success, return 0. On error, errno can+be::++ -EPERM if any of the passed feature are locked.+ -ENOTSUPP if the feature is not supported by the hardware or+ kernel.+ -EINVAL arguments (non existing feature, etc)++The feature's bits supported are::++ ARCH_SHSTK_SHSTK - Shadow stack+ ARCH_SHSTK_WRSS - WRSS++Currently shadow stack and WRSS are supported via this interface. WRSS+can only be enabled with shadow stack, and is automatically disabled+if shadow stack is disabled.++Proc Status+===========+To check if an application is actually running with shadow stack, the+user can read the /proc/$PID/status. It will report "wrss" or "shstk"+depending on what is enabled. The lines look like this::++ x86_Thread_features: shstk wrss+ x86_Thread_features_locked: shstk wrss++Implementation of the Shadow Stack+==================================++Shadow Stack Size+-----------------++A task's shadow stack is allocated from memory to a fixed size of+MIN(RLIMIT_STACK, 4 GB). In other words, the shadow stack is allocated to+the maximum size of the normal stack, but capped to 4 GB. However,+a compat-mode application's address space is smaller, each of its thread's+shadow stack size is MIN(1/4 RLIMIT_STACK, 4 GB).++Signal+------++By default, the main program and its signal handlers use the same shadow+stack. Because the shadow stack stores only return addresses, a large+shadow stack covers the condition that both the program stack and the+signal alternate stack run out.++When a signal happens, the old pre-signal state is pushed on the stack. When+shadow stack is enabled, the shadow stack specific state is pushed onto the+shadow stack. Today this is only the old SSP (shadow stack pointer), pushed+in a special format with bit 63 set. On sigreturn this old SSP token is+verified and restored by the kernel. The kernel will also push the normal+restorer address to the shadow stack to help userspace avoid a shadow stack+violation on the sigreturn path that goes through the restorer.++So the shadow stack signal frame format is as follows::++ |1...old SSP| - Pointer to old pre-signal ssp in sigframe token format+ (bit 63 set to 1)+| ...| - Other state may be added in the future+++32 bit ABI signals are not supported in shadow stack processes. Linux prevents+32 bit execution while shadow stack is enabled by the allocating shadow stack's+outside of the 32 bit address space. When execution enters 32 bit mode, either+via far call or returning to userspace, a #GP is generated by the hardware+which, will be delivered to the process as a segfault. When transitioning to+userspace the register's state will be as if the userspace ip being returned to+caused the segfault.++Fork+----++The shadow stack's vma has VM_SHADOW_STACK flag set; its PTEs are required+to be read-only and dirty. When a shadow stack PTE is not RO and dirty, a+shadow access triggers a page fault with the shadow stack access bit set+in the page fault error code.++When a task forks a child, its shadow stack PTEs are copied and both the+parent's and the child's shadow stack PTEs are cleared of the dirty bit.+Upon the next shadow stack access, the resulting shadow stack page fault+is handled by page copy/re-use.++When a pthread child is created, the kernel allocates a new shadow stack+for the new thread. New shadow stack's behave like mmap() with respect to+ASLR behavior.++Exec+----++On exec, shadow stack features are disabled by the kernel. At which point,+userspace can choose to re-enable, or lock them.
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:16:38
From: Yu-cheng Yu <redacted>
Shadow stack provides protection for applications against function return
address corruption. It is active when the processor supports it, the
kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built
for the feature. This is only implemented for the 64-bit kernel. When it
is enabled, legacy non-shadow stack applications continue to work, but
without protection.
Since there is another feature that utilizes CET (Kernel IBT) that will
share implementation with shadow stacks, create CONFIG_CET to signify
that at least one CET feature is configured.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v5:
- Remove capitalization of shadow stack (Boris)
v3:
- Add X86_CET (Kees)
- Add back WRUSS dependency (Kees)
- Fix verbiage (Dave)
- Change from promt to bool (Kirill)
- Add more to commit log
v2:
- Remove already wrong kernel size increase info (tlgx)
- Change prompt to remove "Intel" (tglx)
- Update line about what CPUs are supported (Dave)
Yu-cheng v25:
- Remove X86_CET and use X86_SHADOW_STACK directly.
---
arch/x86/Kconfig | 24 ++++++++++++++++++++++++
arch/x86/Kconfig.assembler | 5 +++++
2 files changed, 29 insertions(+)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:16:43
From: Yu-cheng Yu <redacted>
The Control-Flow Enforcement Technology contains two related features,
one of which is Shadow Stacks. Future patches will utilize this feature
for shadow stack support in KVM, so add a CPU feature flags for Shadow
Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]).
To protect shadow stack state from malicious modification, the registers
are only accessible in supervisor mode. This implementation
context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend
on XSAVES.
The shadow stack feature, enumerated by the CPUID bit described above,
encompasses both supervisor and userspace support for shadow stack. In
near future patches, only userspace shadow stack will be enabled. In
expectation of future supervisor shadow stack support, create a software
CPU capability to enumerate kernel utilization of userspace shadow stack
support. This user shadow stack bit should depend on the HW "shstk"
capability and that logic will be implemented in future patches.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v5:
- Drop "shstk" from cpuinfo (Boris)
- Remove capitalization on shadow stack (Boris)
v3:
- Add user specific shadow stack cpu cap (Andrew Cooper)
- Drop reviewed-bys from Boris and Kees due to the above change.
v2:
- Remove IBT reference in commit log (Kees)
- Describe xsaves dependency using text from (Dave)
v1:
- Remove IBT, can be added in a follow on IBT series.
---
arch/x86/include/asm/cpufeatures.h | 2 ++
arch/x86/include/asm/disabled-features.h | 8 +++++++-
arch/x86/kernel/cpu/cpuid-deps.c | 1 +
3 files changed, 10 insertions(+), 1 deletion(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:17:05
From: Yu-cheng Yu <redacted>
Setting CR4.CET is a prerequisite for utilizing any CET features, most of
which also require setting MSRs.
Kernel IBT already enables the CET CR4 bit when it detects IBT HW support
and is configured with kernel IBT. However, future patches that enable
userspace shadow stack support will need the bit set as well. So change
the logic to enable it in either case.
Clear MSR_IA32_U_CET in cet_disable() so that it can't live to see
userspace in a new kexec-ed kernel that has CR4.CET set from kernel IBT.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v5:
- Remove #ifdeffery (Boris)
v4:
- Add back dedicated command line disable: "nousershstk" (Boris)
v3:
- Remove stay new line (Boris)
- Simplify commit log (Andrew Cooper)
v2:
- In the shadow stack case, go back to only setting CR4.CET if the
kernel is compiled with user shadow stack support.
- Clear MSR_IA32_U_CET as well. (PeterZ)
---
arch/x86/kernel/cpu/common.c | 35 +++++++++++++++++++++++++++--------
1 file changed, 27 insertions(+), 8 deletions(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:17:18
From: Yu-cheng Yu <redacted>
Shadow stack register state can be managed with XSAVE. The registers
can logically be separated into two groups:
* Registers controlling user-mode operation
* Registers controlling kernel-mode operation
The architecture has two new XSAVE state components: one for each group
of those groups of registers. This lets an OS manage them separately if
it chooses. Future patches for host userspace and KVM guests will only
utilize the user-mode registers, so only configure XSAVE to save
user-mode registers. This state will add 16 bytes to the xsave buffer
size.
Future patches will use the user-mode XSAVE area to save guest user-mode
CET state. However, VMCS includes new fields for guest CET supervisor
states. KVM can use these to save and restore guest supervisor state, so
host supervisor XSAVE support is not required.
Adding this exacerbates the already unwieldy if statement in
check_xstate_against_struct() that handles warning about un-implemented
xfeatures. So refactor these check's by having XCHECK_SZ() set a bool when
it actually check's the xfeature. This ends up exceeding 80 chars, but was
better on balance than other options explored. Pass the bool as pointer to
make it clear that XCHECK_SZ() can change the variable.
While configuring user-mode XSAVE, clarify kernel-mode registers are not
managed by XSAVE by defining the xfeature in
XFEATURE_MASK_SUPERVISOR_UNSUPPORTED, like is done for XFEATURE_MASK_PT.
This serves more of a documentation as code purpose, and functionally,
only enables a few safety checks.
Both XSAVE state components are supervisor states, even the state
controlling user-mode operation. This is a departure from earlier features
like protection keys where the PKRU state is a normal user
(non-supervisor) state. Having the user state be supervisor-managed
ensures there is no direct, unprivileged access to it, making it harder
for an attacker to subvert CET.
To facilitate this privileged access, define the two user-mode CET MSRs,
and the bits defined in those MSRs relevant to future shadow stack
enablement patches.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v5:
- Move comments from end of lines in cet_user_state struct (Boris)
v3:
- Add missing "is" in commit log (Boris)
- Change to case statement for struct size checking (Boris)
- Adjust commas on xfeature_names (Kees, Boris)
v2:
- Change name to XFEATURE_CET_KERNEL_UNUSED (peterz)
KVM refresh:
- Reword commit log using some verbiage posted by Dave Hansen
- Remove unlikely to be used supervisor cet xsave struct
- Clarify that supervisor cet state is not saved by xsave
- Remove unused supervisor MSRs
---
arch/x86/include/asm/fpu/types.h | 16 +++++-
arch/x86/include/asm/fpu/xstate.h | 6 ++-
arch/x86/kernel/fpu/xstate.c | 90 +++++++++++++++----------------
3 files changed, 61 insertions(+), 51 deletions(-)
@@ -50,7 +50,8 @@#define XFEATURE_MASK_USER_DYNAMIC XFEATURE_MASK_XTILE_DATA/* All currently supported supervisor features */-#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID)+#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID | \+XFEATURE_MASK_CET_USER)/**Asupervisorstatecomponentmaynotalwayscontainvaluableinformation,
@@ -77,7 +78,8 @@*Unsupportedsupervisorfeatures.Whenasupervisorfeatureinthismaskis*supportedinthefuture,moveittothesupportedsupervisorfeaturemask.*/-#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT)+#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT | \+XFEATURE_MASK_CET_KERNEL)/* All supervisor states including supported and unsupported states. */#define XFEATURE_MASK_SUPERVISOR_ALL (XFEATURE_MASK_SUPERVISOR_SUPPORTED | \
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:17:22
Just like user xfeatures, supervisor xfeatures can be active in the
registers or present in the task FPU buffer. If the registers are
active, the registers can be modified directly. If the registers are
not active, the modification must be performed on the task FPU buffer.
When the state is not active, the kernel could perform modifications
directly to the buffer. But in order for it to do that, it needs
to know where in the buffer the specific state it wants to modify is
located. Doing this is not robust against optimizations that compact
the FPU buffer, as each access would require computing where in the
buffer it is.
The easiest way to modify supervisor xfeature data is to force restore
the registers and write directly to the MSRs. Often times this is just fine
anyway as the registers need to be restored before returning to userspace.
Do this for now, leaving buffer writing optimizations for the future.
Add a new function fpregs_lock_and_load() that can simultaneously call
fpregs_lock() and do this restore. Also perform some extra sanity
checks in this function since this will be used in non-fpu focused code.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Suggested-by: Thomas Gleixner <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Drop "but appear to work" (Boris)
v5:
- Fix spelling error (Boris)
- Don't export fpregs_lock_and_load() (Boris)
v3:
- Rename to fpregs_lock_and_load() to match the unlocking
fpregs_unlock(). (Kees)
- Elaborate in comment about helper. (Dave)
v2:
- Drop optimization of writing directly the buffer, and change API
accordingly.
- fpregs_lock_and_load() suggested by tglx
- Some commit log verbiage from dhansen
---
arch/x86/include/asm/fpu/api.h | 9 +++++++++
arch/x86/kernel/fpu/core.c | 18 ++++++++++++++++++
2 files changed, 27 insertions(+)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:17:46
Today the control protection handler is defined in traps.c and used only
for the kernel IBT feature. To reduce ifdeffery, move it to it's own file.
In future patches, functionality will be added to make this handler also
handle user shadow stack faults. So name the file cet.c.
No functional change.
Tested-by: Pengfei Xu <redacted>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Split move to cet.c and shadow stack enhancements to fault handler to
separate files. (Kees)
---
arch/x86/kernel/Makefile | 2 ++
arch/x86/kernel/cet.c | 76 ++++++++++++++++++++++++++++++++++++++++
arch/x86/kernel/traps.c | 75 ---------------------------------------
3 files changed, 78 insertions(+), 75 deletions(-)
create mode 100644 arch/x86/kernel/cet.c
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:17:58
From: Yu-cheng Yu <redacted>
New processors that support Shadow Stack regard Write=0,Dirty=1 PTEs as
shadow stack pages.
In normal cases, it can be helpful to create Write=1 PTEs as also Dirty=1
if HW dirty tracking is not needed, because if the Dirty bit is not already
set the CPU has to set Dirty=1 when the memory gets written to. This
creates additional work for the CPU. So traditional wisdom was to simply
set the Dirty bit whenever you didn't care about it. However, it was never
really very helpful for read-only kernel memory.
When CR4.CET=1 and IA32_S_CET.SH_STK_EN=1, some instructions can write to
such supervisor memory. The kernel does not set IA32_S_CET.SH_STK_EN, so
avoiding kernel Write=0,Dirty=1 memory is not strictly needed for any
functional reason. But having Write=0,Dirty=1 kernel memory doesn't have
any functional benefit either, so to reduce ambiguity between shadow stack
and regular Write=0 pages, remove Dirty=1 from any kernel Write=0 PTEs.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Kees Cook <redacted>
Cc: Thomas Gleixner <redacted>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Peter Zijlstra <peterz@infradead.org>
---
v6:
- Also remove dirty from newly added set_memory_rox()
v5:
- Spelling and grammer in commit log (Boris)
v3:
- Update commit log (Andrew Cooper, Peterz)
v2:
- Normalize PTE bit descriptions between patches
---
arch/x86/include/asm/pgtable_types.h | 6 +++---
arch/x86/mm/pat/set_memory.c | 4 ++--
2 files changed, 5 insertions(+), 5 deletions(-)
@@ -2073,12 +2073,12 @@ int set_memory_nx(unsigned long addr, int numpages)intset_memory_ro(unsignedlongaddr,intnumpages){-returnchange_page_attr_clear(&addr,numpages,__pgprot(_PAGE_RW),0);+returnchange_page_attr_clear(&addr,numpages,__pgprot(_PAGE_RW|_PAGE_DIRTY),0);}intset_memory_rox(unsignedlongaddr,intnumpages){-pgprot_tclr=__pgprot(_PAGE_RW);+pgprot_tclr=__pgprot(_PAGE_RW|_PAGE_DIRTY);if(__supported_pte_mask&_PAGE_NX)clr.pgprot|=_PAGE_NX;
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:18:01
From: Yu-cheng Yu <redacted>
A control-protection fault is triggered when a control-flow transfer
attempt violates Shadow Stack or Indirect Branch Tracking constraints.
For example, the return address for a RET instruction differs from the copy
on the shadow stack.
There already exists a control-protection fault handler for handling kernel
IBT faults. Refactor this fault handler into separate user and kernel
handlers, like the page fault handler. Add a control-protection handler
for usermode. To avoid ifdeffery, put them both in a new file cet.c, which
is compiled in the case of either of the two CET features supported in the
kernel: kernel IBT or user mode shadow stack. Move some static inline
functions from traps.c into a header so they can be used in cet.c.
Opportunistically fix a comment in the kernel IBT part of the fault
handler that is on the end of the line instead of preceding it.
Keep the same behavior for the kernel side of the fault handler, except for
converting a BUG to a WARN in the case of a #CP happening when the feature
is missing. This unifies the behavior with the new shadow stack code, and
also prevents the kernel from crashing under this situation which is
potentially recoverable.
The control-protection fault handler works in a similar way as the general
protection fault handler. It provides the si_code SEGV_CPERR to the signal
handler.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
Cc: Michael Kerrisk <redacted>
---
v6:
- Split into separate patches (Kees)
- Change to "x86/shstk" in commit log (Boris)
v5:
- Move to separate file to advoid ifdeffery (Boris)
- Improvements to commit log (Boris)
- Rename control_protection_err (Boris)
- Move comment from end of line in IBT fault handler (Boris)
v3:
- Shorten user/kernel #CP handler function names (peterz)
- Restore CP_ENDBR check to kernel handler (peterz)
- Utilize CONFIG_X86_CET (Kees)
- Unify "unexpected" warnings (Andrew Cooper)
- Use 2d array for error code chars (Andrew Cooper)
- Add comment about why to read SSP MSR before enabling interrupts
v2:
- Integrate with kernel IBT fault handler
- Update printed messages. (Dave)
- Remove array_index_nospec() usage. (Dave)
- Remove IBT messages. (Dave)
- Add enclave error code bit processing it case it can get triggered
somehow.
- Add extra "unknown" in control_protection_err.
---
arch/arm/kernel/signal.c | 2 +-
arch/arm64/kernel/signal.c | 2 +-
arch/arm64/kernel/signal32.c | 2 +-
arch/sparc/kernel/signal32.c | 2 +-
arch/sparc/kernel/signal_64.c | 2 +-
arch/x86/include/asm/disabled-features.h | 8 +-
arch/x86/include/asm/idtentry.h | 2 +-
arch/x86/include/asm/traps.h | 12 +++
arch/x86/kernel/cet.c | 94 +++++++++++++++++++++---
arch/x86/kernel/idt.c | 2 +-
arch/x86/kernel/signal_32.c | 2 +-
arch/x86/kernel/signal_64.c | 2 +-
arch/x86/kernel/traps.c | 12 ---
arch/x86/xen/enlighten_pv.c | 2 +-
arch/x86/xen/xen-asm.S | 2 +-
include/uapi/asm-generic/siginfo.h | 3 +-
16 files changed, 117 insertions(+), 34 deletions(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:18:32
From: Yu-cheng Yu <redacted>
To prepare the introduction of _PAGE_SAVED_DIRTY, move pmd_write() and
pud_write() up in the file, so that they can be used by other
helpers below. No functional changes.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Reviewed-by: Kirill A. Shutemov <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
arch/x86/include/asm/pgtable.h | 24 ++++++++++++------------
1 file changed, 12 insertions(+), 12 deletions(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:18:52
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One of these changes is to allow for pte_mkwrite() to create different
types of writable memory (the existing conventionally writable type and
also the new shadow stack type). Future patches will convert pte_mkwrite()
to take a VMA in order to facilitate this, however there are places in the
kernel where pte_mkwrite() is called outside of the context of a VMA.
These are for kernel memory. So create a new variant called
pte_mkwrite_kernel() and switch the kernel users over to it. Have
pte_mkwrite() and pte_mkwrite_kernel() be the same for now. Future patches
will introduce changes to make pte_mkwrite() take a VMA.
Only do this for architectures that need it because they call pte_mkwrite()
in arch code without an associated VMA. Since it will only currently be
used in arch code, so do not include it in arch_pgtable_helpers.rst.
Cc: linux-doc@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-s390@vger.kernel.org
Cc: xen-devel@lists.xenproject.org
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Tested-by: Pengfei Xu <redacted>
Suggested-by: David Hildenbrand <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
Hi Non-x86 Arch’s,
x86 has a feature that allows for the creation of a special type of
writable memory (shadow stack) that is only writable in limited specific
ways. Previously, changes were proposed to core MM code to teach it to
decide when to create normally writable memory or the special shadow stack
writable memory, but David Hildenbrand suggested[0] to change
pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be
moved into x86 code.
Since pXX_mkwrite() is defined in every arch, it requires some tree-wide
changes. So that is why you are seeing some patches out of a big x86
series pop up in your arch mailing list. There is no functional change.
After this refactor, the shadow stack series goes on to use the arch
helpers to push shadow stack memory details inside arch/x86.
Testing was just 0-day build testing.
Hopefully that is enough context. Thanks!
[0] https://lore.kernel.org/lkml/0e29a2d0-08d8-bcd6-ff26-4bea0e4037b0@redhat.com/#t
v6:
- New patch
---
arch/arm64/include/asm/pgtable.h | 7 ++++++-
arch/arm64/mm/trans_pgd.c | 4 ++--
arch/s390/include/asm/pgtable.h | 7 ++++++-
arch/s390/mm/pageattr.c | 2 +-
arch/x86/include/asm/pgtable.h | 7 ++++++-
arch/x86/xen/mmu_pv.c | 2 +-
6 files changed, 22 insertions(+), 7 deletions(-)
@@ -96,7 +96,7 @@ static int walk_pte_level(pmd_t *pmdp, unsigned long addr, unsigned long end,if(flags&SET_MEMORY_RO)new=pte_wrprotect(new);elseif(flags&SET_MEMORY_RW)-new=pte_mkwrite(pte_mkdirty(new));+new=pte_mkwrite_kernel(pte_mkdirty(new));if(flags&SET_MEMORY_NX)new=set_pte_bit(new,__pgprot(_PAGE_NOEXEC));elseif(flags&SET_MEMORY_X)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:18:59
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One of these changes is to allow for pmd_mkwrite() to create different
types of writable memory (the existing conventionally writable type and
also the new shadow stack type). Future patches will convert pmd_mkwrite()
to take a VMA in order to facilitate this, however there are places in the
kernel where pmd_mkwrite() is called outside of the context of a VMA.
These are for kernel memory. So create a new variant called
pmd_mkwrite_kernel() and switch the kernel users over to it. Have
pmd_mkwrite() and pmd_mkwrite_kernel() be the same for now. Future patches
will introduce changes to make pmd_mkwrite() take a VMA.
Only do this for architectures that need it because they call pmd_mkwrite()
in arch code without an associated VMA. Since it will only currently be
used in arch code, so do not include it in arch_pgtable_helpers.rst.
Cc: linux-kernel@vger.kernel.org
Cc: linux-s390@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Tested-by: Pengfei Xu <redacted>
Suggested-by: David Hildenbrand <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
Hi Non-x86 Arch’s,
x86 has a feature that allows for the creation of a special type of
writable memory (shadow stack) that is only writable in limited specific
ways. Previously, changes were proposed to core MM code to teach it to
decide when to create normally writable memory or the special shadow stack
writable memory, but David Hildenbrand suggested[0] to change
pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be
moved into x86 code.
Since pXX_mkwrite() is defined in every arch, it requires some tree-wide
changes. So that is why you are seeing some patches out of a big x86
series pop up in your arch mailing list. There is no functional change.
After this refactor, the shadow stack series goes on to use the arch
helpers to push shadow stack memory details inside arch/x86.
Testing was just 0-day build testing.
Hopefully that is enough context. Thanks!
[0] https://lore.kernel.org/lkml/0e29a2d0-08d8-bcd6-ff26-4bea0e4037b0@redhat.com/#t
v6:
- New patch
---
arch/s390/include/asm/pgtable.h | 7 ++++++-
arch/s390/mm/pageattr.c | 2 +-
2 files changed, 7 insertions(+), 2 deletions(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:19:11
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One of these unusual properties is that shadow stack memory is writable,
but only in limited ways. These limits are applied via a specific PTE
bit combination. Nevertheless, the memory is writable, and core mm code
will need to apply the writable permissions in the typical paths that
call pte_mkwrite().
In addition to VM_WRITE, the shadow stack VMA's will have a flag denoting
that they are special shadow stack flavor of writable memory. So make
pte_mkwrite() take a VMA, so that the x86 implementation of it can know to
create regular writable memory or shadow stack memory.
Apply the same changes for pmd_mkwrite() and huge_pte_mkwrite().
No functional change.
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-alpha@vger.kernel.org
Cc: linux-snps-arc@lists.infradead.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-csky@vger.kernel.org
Cc: linux-hexagon@vger.kernel.org
Cc: linux-ia64@vger.kernel.org
Cc: loongarch@lists.linux.dev
Cc: linux-m68k@lists.linux-m68k.org
Cc: Michal Simek <monstr@monstr.eu>
Cc: Dinh Nguyen <dinguyen@kernel.org>
Cc: linux-mips@vger.kernel.org
Cc: openrisc@lists.librecores.org
Cc: linux-parisc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-riscv@lists.infradead.org
Cc: linux-s390@vger.kernel.org
Cc: linux-sh@vger.kernel.org
Cc: sparclinux@vger.kernel.org
Cc: linux-um@lists.infradead.org
Cc: xen-devel@lists.xenproject.org
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Tested-by: Pengfei Xu <redacted>
Suggested-by: David Hildenbrand <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
Hi Non-x86 Arch’s,
x86 has a feature that allows for the creation of a special type of
writable memory (shadow stack) that is only writable in limited specific
ways. Previously, changes were proposed to core MM code to teach it to
decide when to create normally writable memory or the special shadow stack
writable memory, but David Hildenbrand suggested[0] to change
pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be
moved into x86 code.
Since pXX_mkwrite() is defined in every arch, it requires some tree-wide
changes. So that is why you are seeing some patches out of a big x86
series pop up in your arch mailing list. There is no functional change.
After this refactor, the shadow stack series goes on to use the arch
helpers to push shadow stack memory details inside arch/x86.
Testing was just 0-day build testing.
Hopefully that is enough context. Thanks!
[0] https://lore.kernel.org/lkml/0e29a2d0-08d8-bcd6-ff26-4bea0e4037b0@redhat.com/#t
v6:
- New patch
---
Documentation/mm/arch_pgtable_helpers.rst | 9 ++++++---
arch/alpha/include/asm/pgtable.h | 6 +++++-
arch/arc/include/asm/hugepage.h | 2 +-
arch/arc/include/asm/pgtable-bits-arcv2.h | 7 ++++++-
arch/arm/include/asm/pgtable-3level.h | 7 ++++++-
arch/arm/include/asm/pgtable.h | 2 +-
arch/arm64/include/asm/pgtable.h | 4 ++--
arch/csky/include/asm/pgtable.h | 2 +-
arch/hexagon/include/asm/pgtable.h | 2 +-
arch/ia64/include/asm/pgtable.h | 2 +-
arch/loongarch/include/asm/pgtable.h | 4 ++--
arch/m68k/include/asm/mcf_pgtable.h | 2 +-
arch/m68k/include/asm/motorola_pgtable.h | 6 +++++-
arch/m68k/include/asm/sun3_pgtable.h | 6 +++++-
arch/microblaze/include/asm/pgtable.h | 2 +-
arch/mips/include/asm/pgtable.h | 6 +++---
arch/nios2/include/asm/pgtable.h | 2 +-
arch/openrisc/include/asm/pgtable.h | 2 +-
arch/parisc/include/asm/pgtable.h | 6 +++++-
arch/powerpc/include/asm/book3s/32/pgtable.h | 2 +-
arch/powerpc/include/asm/book3s/64/pgtable.h | 4 ++--
arch/powerpc/include/asm/nohash/32/pgtable.h | 2 +-
arch/powerpc/include/asm/nohash/32/pte-8xx.h | 2 +-
arch/powerpc/include/asm/nohash/64/pgtable.h | 2 +-
arch/riscv/include/asm/pgtable.h | 6 +++---
arch/s390/include/asm/hugetlb.h | 4 ++--
arch/s390/include/asm/pgtable.h | 4 ++--
arch/sh/include/asm/pgtable_32.h | 10 ++++++++--
arch/sparc/include/asm/pgtable_32.h | 2 +-
arch/sparc/include/asm/pgtable_64.h | 6 +++---
arch/um/include/asm/pgtable.h | 2 +-
arch/x86/include/asm/pgtable.h | 6 ++++--
arch/xtensa/include/asm/pgtable.h | 2 +-
include/asm-generic/hugetlb.h | 4 ++--
include/linux/mm.h | 2 +-
mm/debug_vm_pgtable.c | 16 ++++++++--------
mm/huge_memory.c | 6 +++---
mm/hugetlb.c | 4 ++--
mm/memory.c | 4 ++--
mm/migrate_device.c | 2 +-
mm/mprotect.c | 2 +-
mm/userfaultfd.c | 2 +-
42 files changed, 106 insertions(+), 69 deletions(-)
@@ -85,7 +85,7 @@#ifndef __ASSEMBLY__/* pte_clear moved to later in this file */-staticinlinepte_tpte_mkwrite(pte_tpte)+staticinlinepte_tpte_mkwrite(pte_tpte,structvm_area_struct*vma){return__pte(pte_val(pte)|_PAGE_RW);}
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:19:30
Some OSes have a greater dependence on software available bits in PTEs than
Linux. That left the hardware architects looking for a way to represent a
new memory type (shadow stack) within the existing bits. They chose to
repurpose a lightly-used state: Write=0,Dirty=1. So in order to support
shadow stack memory, Linux should avoid creating memory with this PTE bit
combination unless it intends for it to be shadow stack.
The reason it's lightly used is that Dirty=1 is normally set by HW
_before_ a write. A write with a Write=0 PTE would typically only generate
a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and*
generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which
supports shadow stacks will no longer exhibit this oddity.
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this,
in places where Linux normally creates Write=0,Dirty=1, it can use the
software-defined _PAGE_SAVED_DIRTY in place of the hardware _PAGE_DIRTY.
In other words, whenever Linux needs to create Write=0,Dirty=1, it instead
creates Write=0,SavedDirty=1 except for shadow stack, which is
Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit
combinations would be set as follows for various types of memory:
(Write=0,SavedDirty=1,Dirty=0):
- A modified, copy-on-write (COW) page. Previously when a typical
anonymous writable mapping was made COW via fork(), the kernel would
mark it Write=0,Dirty=1. Now it will instead use the SavedDirty bit.
This happens in copy_present_pte().
- A R/O page that has been COW'ed. The user page is in a R/O VMA,
and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault
handler creates a copy of the page and sets the new copy's PTE as
Write=0 and SavedDirty=1.
- A shared shadow stack PTE. When a shadow stack page is being shared
among processes (this happens at fork()), its PTE is made Dirty=0, so
the next shadow stack access causes a fault, and the page is
duplicated and Dirty=1 is set again. This is the COW equivalent for
shadow stack pages, even though it's copy-on-access rather than
copy-on-write.
(Write=0,SavedDirty=0,Dirty=1):
- A shadow stack PTE.
- A Cow PTE created when a processor without shadow stack support set
Dirty=1.
There are six bits left available to software in the 64-bit PTE after
consuming a bit for _PAGE_SAVED_DIRTY. No space is consumed in 32-bit
kernels because shadow stacks are not enabled there.
Implement only the infrastructure for _PAGE_SAVED_DIRTY. Changes to start
creating _PAGE_SAVED_DIRTY PTEs will follow once other pieces are in place.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Co-developed-by: Yu-cheng Yu <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Rename _PAGE_COW to _PAGE_SAVED_DIRTY (David Hildenbrand)
- Add _PAGE_SAVED_DIRTY to _PAGE_CHG_MASK
v5:
- Fix log, comments and whitespace (Boris)
- Remove capitalization on shadow stack (Boris)
v4:
- Teach pte_flags_need_flush() about _PAGE_COW bit
- Break apart patch for better bisectability
v3:
- Add comment around _PAGE_TABLE in response to comment
from (Andrew Cooper)
- Check for PSE in pmd_shstk (Andrew Cooper)
- Get to the point quicker in commit log (Andrew Cooper)
- Clarify and reorder commit log for why the PTE bit examples have
multiple entries. Apply same changes for comment. (peterz)
- Fix comment that implied dirty bit for COW was a specific x86 thing
(peterz)
- Fix swapping of Write/Dirty (PeterZ)
---
arch/x86/include/asm/pgtable.h | 79 ++++++++++++++++++++++++++++
arch/x86/include/asm/pgtable_types.h | 65 ++++++++++++++++++++---
arch/x86/include/asm/tlbflush.h | 3 +-
3 files changed, 138 insertions(+), 9 deletions(-)
@@ -21,7 +21,8 @@#define _PAGE_BIT_SOFTW2 10 /* " */#define _PAGE_BIT_SOFTW3 11 /* " */#define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */-#define _PAGE_BIT_SOFTW4 58 /* available for programmer */+#define _PAGE_BIT_SOFTW4 57 /* available for programmer */+#define _PAGE_BIT_SOFTW5 58 /* available for programmer */#define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */#define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */#define _PAGE_BIT_PKEY_BIT2 61 /* Protection Keys, bit 3/4 */
@@ -34,6 +35,15 @@#define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */#define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4+/*+*IndicatesaSavedDirtybitpage.+*/+#ifdef CONFIG_X86_USER_SHADOW_STACK+#define _PAGE_BIT_SAVED_DIRTY _PAGE_BIT_SOFTW5 /* copy-on-write */+#else+#define _PAGE_BIT_SAVED_DIRTY 0+#endif+/* If _PAGE_BIT_PRESENT is clear, we use these: *//* - if the user mapped it with PROT_NONE; pte_present gives true */#define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:19:42
From: Yu-cheng Yu <redacted>
When shadow stack is in use, Write=0,Dirty=1 PTE are preserved for
shadow stack. Copy-on-write PTEs then have Write=0,SavedDirty=1.
When a PTE goes from Write=1,Dirty=1 to Write=0,SavedDirty=1, it could
become a transient shadow stack PTE in two cases:
1. Some processors can start a write but end up seeing a Write=0 PTE by
the time they get to the Dirty bit, creating a transient shadow stack
PTE. However, this will not occur on processors supporting shadow
stack, and a TLB flush is not necessary.
2. When _PAGE_DIRTY is replaced with _PAGE_SAVED_DIRTY non-atomically, a
transient shadow stack PTE can be created as a result. Thus, prevent
that with cmpxchg.
In the case of pmdp_set_wrprotect(), for nopmd configs the ->pmd operated
on does not exist and the logic would need to be different. Although the
extra functionality will normally be optimized out when user shadow
stacks are not configured, also exclude it in the preprocessor stage so
that it will still compile. User shadow stack is not supported there by
Linux anyway. Leave the cpu_feature_enabled() check so that the
functionality also gets disabled based on runtime detection of the
feature.
Similarly, compile it out in ptep_set_wrprotect() due to a clang warning
on i386. Like above, the code path should get optimized out on i386
since shadow stack is not supported on 32 bit kernels, but this makes
the compiler happy.
Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many
insights to the issue. Jann Horn provided the cmpxchg solution.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Fix comment and log to update for _PAGE_COW being replaced with
_PAGE_SAVED_DIRTY.
v5:
- Commit log verbiage and formatting (Boris)
- Remove capitalization on shadow stack (Boris)
- Fix i386 warning on recent clang
v3:
- Remove unnecessary #ifdef (Dave Hansen)
v2:
- Compile out some code due to clang build error
- Clarify commit log (dhansen)
- Normalize PTE bit descriptions between patches (dhansen)
- Update comment with text from (dhansen)
---
arch/x86/include/asm/pgtable.h | 35 ++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:19:45
The recently introduced _PAGE_SAVED_DIRTY should be used instead of the
HW Dirty bit whenever a PTE is Write=0, in order to not inadvertently
create shadow stack PTEs. Update pte_mk*() helpers to do this, and apply
the same changes to pmd and pud.
For pte_modify() this is a bit trickier. It takes a "raw" pgprot_t which
was not necessarily created with any of the existing PTE bit helpers.
That means that it can return a pte_t with Write=0,Dirty=1, a shadow
stack PTE, when it did not intend to create one.
Modify it to also move _PAGE_DIRTY to _PAGE_SAVED_DIRTY. To avoid
creating Write=0,Dirty=1 PTEs, pte_modify() needs to avoid:
1. Marking Write=0 PTEs Dirty=1
2. Marking Dirty=1 PTEs Write=0
The first case cannot happen as the existing behavior of pte_modify() is to
filter out any Dirty bit passed in newprot. Handle the second case by
shifting _PAGE_DIRTY=1 to _PAGE_SAVED_DIRTY=1 if the PTE was write
protected by the pte_modify() call. Apply the same changes to
pmd_modify().
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Co-developed-by: Yu-cheng Yu <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Rename _PAGE_COW to _PAGE_SAVED_DIRTY (David Hildenbrand)
- Open code _PAGE_SAVED_DIRTY part in pte_modify() (Boris)
- Change the logic so the open coded part is not too ugly
- Merge pte_modify() patch with this one because of the above
v4:
- Break part patch for better bisectability
---
arch/x86/include/asm/pgtable.h | 168 ++++++++++++++++++++++++++++-----
1 file changed, 145 insertions(+), 23 deletions(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:20:06
From: Yu-cheng Yu <redacted>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
Future patches will introduce a new VM flag VM_SHADOW_STACK that will be
VM_HIGH_ARCH_BIT_5. VM_HIGH_ARCH_BIT_1 through VM_HIGH_ARCH_BIT_4 are
bits 32-36, and bit 37 is the unrelated VM_UFFD_MINOR_BIT. For the sake
of order, make all VM_HIGH_ARCH_BITs stay together by moving
VM_UFFD_MINOR_BIT from 37 to 38. This will allow VM_SHADOW_STACK to be
introduced as 37.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Acked-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Yu-cheng Yu <redacted>
Reviewed-by: Axel Rasmussen <axelrasmussen@google.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Mike Kravetz <redacted>
---
include/linux/mm.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:20:11
From: Yu-cheng Yu <redacted>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
A shadow stack PTE must be read-only and have _PAGE_DIRTY set. However,
read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These
two cases are handled differently for page faults. Introduce
VM_SHADOW_STACK to track shadow stack VMAs.
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Yu-cheng Yu <redacted>
Reviewed-by: Kirill A. Shutemov <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v6:
- Add comment about VM_SHADOW_STACK not being allowed with VM_SHARED
(David Hildenbrand)
v3:
- Drop arch specific change in arch_vma_name(). The memory can show as
anonymous (Kirill)
- Change CONFIG_ARCH_HAS_SHADOW_STACK to CONFIG_X86_USER_SHADOW_STACK
in show_smap_vma_flags() (Boris)
---
Documentation/filesystems/proc.rst | 1 +
fs/proc/task_mmu.c | 3 +++
include/linux/mm.h | 8 ++++++++
3 files changed, 12 insertions(+)
@@ -564,6 +564,7 @@ encoded manner. The codes are the following: mt arm64 MTE allocation tags are enabled um userfaultfd missing tracking uw userfaultfd wr-protect tracking+ ss shadow stack page == ======================================= Note that there is no guarantee that every flag and associated mnemonic will
@@ -315,11 +315,13 @@ extern unsigned int kobjsize(const void *objp);#define VM_HIGH_ARCH_BIT_2 34 /* bit only usable on 64-bit architectures */#define VM_HIGH_ARCH_BIT_3 35 /* bit only usable on 64-bit architectures */#define VM_HIGH_ARCH_BIT_4 36 /* bit only usable on 64-bit architectures */+#define VM_HIGH_ARCH_BIT_5 37 /* bit only usable on 64-bit architectures */#define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0)#define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1)#define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2)#define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3)#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4)+#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5)#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */#ifdef CONFIG_ARCH_HAS_PKEYS
@@ -335,6 +337,12 @@ extern unsigned int kobjsize(const void *objp);#endif#endif /* CONFIG_ARCH_HAS_PKEYS */+#ifdef CONFIG_X86_USER_SHADOW_STACK+# define VM_SHADOW_STACK VM_HIGH_ARCH_5 /* Should not be set with VM_SHARED */+#else+# define VM_SHADOW_STACK VM_NONE+#endif+#if defined(CONFIG_X86)# define VM_PAT VM_ARCH_1 /* PAT reserves whole VMA at once (x86) */#elif defined(CONFIG_PPC)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:20:16
From: Yu-cheng Yu <redacted>
The CPU performs "shadow stack accesses" when it expects to encounter
shadow stack mappings. These accesses can be implicit (via CALL/RET
instructions) or explicit (instructions like WRSS).
Shadow stack accesses to shadow-stack mappings can result in faults in
normal, valid operation just like regular accesses to regular mappings.
Shadow stacks need some of the same features like delayed allocation, swap
and copy-on-write. The kernel needs to use faults to implement those
features.
The architecture has concepts of both shadow stack reads and shadow stack
writes. Any shadow stack access to non-shadow stack memory will generate
a fault with the shadow stack error code bit set.
This means that, unlike normal write protection, the fault handler needs
to create a type of memory that can be written to (with instructions that
generate shadow stack writes), even to fulfill a read access. So in the
case of COW memory, the COW needs to take place even with a shadow stack
read. Otherwise the page will be left (shadow stack) writable in
userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE
for shadow stack accesses, even if the access was a shadow stack read.
For the purpose of making this clearer, consider the following example.
If a process has a shadow stack, and forks, the shadow stack PTEs will
become read-only due to COW. If the CPU in one process performs a shadow
stack read access to the shadow stack, for example executing a RET and
causing the CPU to read the shadow stack copy of the return address, then
in order for the fault to be resolved the PTE will need to be set with
shadow stack permissions. But then the memory would be changeable from
userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger
COW, otherwise the shared page would be changeable from both processes.
Shadow stack accesses can also result in errors, such as when a shadow
stack overflows, or if a shadow stack access occurs to a non-shadow-stack
mapping. Also, generate the errors for invalid shadow stack accesses.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Update comment due to rename of Cow bit to SavedDirty
v5:
- Add description of COW example (Boris)
- Replace "permissioned" (Boris)
- Remove capitalization of shadow stack (Boris)
v4:
- Further improve comment talking about FAULT_FLAG_WRITE (Peterz)
v3:
- Improve comment talking about using FAULT_FLAG_WRITE (Peterz)
---
arch/x86/include/asm/trap_pf.h | 2 ++
arch/x86/mm/fault.c | 38 ++++++++++++++++++++++++++++++++++
2 files changed, 40 insertions(+)
@@ -1138,8 +1138,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma)(error_code&X86_PF_INSTR),foreign))return1;+/*+*Shadowstackaccesses(PF_SHSTK=1)areonlypermittedto+*shadowstackVMAs.Allotheraccessesresultinanerror.+*/+if(error_code&X86_PF_SHSTK){+if(unlikely(!(vma->vm_flags&VM_SHADOW_STACK)))+return1;+if(unlikely(!(vma->vm_flags&VM_WRITE)))+return1;+return0;+}+if(error_code&X86_PF_WRITE){/* write, present and write, not present: */+if(unlikely(vma->vm_flags&VM_SHADOW_STACK))+return1;if(unlikely(!(vma->vm_flags&VM_WRITE)))return1;return0;
@@ -1331,6 +1345,30 @@ void do_user_addr_fault(struct pt_regs *regs,perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS,1,regs,address);+/*+*WhenapagebecomesCOWitchangesfromashadowstackpermission+*page(Write=0,Dirty=1)to(Write=0,Dirty=0,SavedDirty=1),whichissimply+*read-onlytotheCPU.Whenshadowstackisenabled,aRETwould+*normallypoptheshadowstackbyreadingitwitha"shadow stack+*read" access. However, in the COW case the shadow stack memory does+*nothaveshadowstackpermissions,itisread-only.Soitwill+*generateafault.+*+*Forconventionallywritablepages,areadcanbeservicedwitha+*readonlyPTE,andCOWwouldnothavetohappen.Butforshadow+*stack,thereisn'ttheconceptofread-onlyshadowstackmemory.+*Ifitisshadowstackpermission,itcanbemodifiedviaCALLand+*RETinstructions.SoCOWneedstohappenbeforeanymemorycanbe+*mappedwithshadowstackpermissions.+*+*Shadowstackaccesses(readorwrite)needtobeservicedwith+*shadowstackpermissionmemory,sointhecaseofashadowstack+*readaccess,treatitasaWRITEfaultsobothCOWwillhappenand+*thewritefaultpathwillticklemaybe_mkwrite()andmapthememory+*shadowstack.+*/+if(error_code&X86_PF_SHSTK)+flags|=FAULT_FLAG_WRITE;if(error_code&X86_PF_WRITE)flags|=FAULT_FLAG_WRITE;if(error_code&X86_PF_INSTR)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:20:35
If a VMA has the VM_SHADOW_STACK flag, it is shadow stack memory. So
when it is made writable with pte_mkwrite(), it should create shadow
stack memory, not conventionally writable memory. Now that pte_mkwrite()
takes a VMA, and places where shadow stack memory might be created pass
one, pte_mkwrite() can know when it should do this.
So make pte_mkwrite() create shadow stack memory when the VMA has the
VM_SHADOW_STACK flag. Do the same thing for pmd_mkwrite().
This requires referencing VM_SHADOW_STACK in these functions, which are
currently defined in pgtable.h, however mm.h (where VM_SHADOW_STACK is
located) can't be pulled in without causing problems for files that
reference pgtable.h. So also move pte/pmd_mkwrite() into pgtable.c, where
they can safely reference VM_SHADOW_STACK.
Tested-by: Pengfei Xu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- New patch
---
arch/x86/include/asm/pgtable.h | 20 ++------------------
arch/x86/mm/pgtable.c | 26 ++++++++++++++++++++++++++
2 files changed, 28 insertions(+), 18 deletions(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:20:38
From: Yu-cheng Yu <redacted>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
The architecture of shadow stack constrains the ability of userspace to
move the shadow stack pointer (SSP) in order to prevent corrupting or
switching to other shadow stacks. The RSTORSSP can move the ssp to
different shadow stacks, but it requires a specially placed token in order
to do this. However, the architecture does not prevent incrementing the
stack pointer to wander onto an adjacent shadow stack. To prevent this in
software, enforce guard pages at the beginning of shadow stack vmas, such
that there will always be a gap between adjacent shadow stacks.
Make the gap big enough so that no userspace SSP changing operations
(besides RSTORSSP), can move the SSP from one stack to the next. The
SSP can increment or decrement by CALL, RET and INCSSP. CALL and RET
can move the SSP by a maximum of 8 bytes, at which point the shadow
stack would be accessed.
The INCSSP instruction can also increment the shadow stack pointer. It
is the shadow stack analog of an instruction like:
addq $0x80, %rsp
However, there is one important difference between an ADD on %rsp and
INCSSP. In addition to modifying SSP, INCSSP also reads from the memory
of the first and last elements that were "popped". It can be thought of
as acting like this:
READ_ONCE(ssp); // read+discard top element on stack
ssp += nr_to_pop * 8; // move the shadow stack
READ_ONCE(ssp-8); // read+discard last popped stack element
The maximum distance INCSSP can move the SSP is 2040 bytes, before it
would read the memory. Therefore a single page gap will be enough to
prevent any operation from shifting the SSP to an adjacent stack, since
it would have to land in the gap at least once, causing a fault.
This could be accomplished by using VM_GROWSDOWN, but this has a
downside. The behavior would allow shadow stack's to grow, which is
unneeded and adds a strange difference to how most regular stacks work.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v5:
- Fix typo in commit log
v4:
- Drop references to 32 bit instructions
- Switch to generic code to drop __weak (Peterz)
v2:
- Use __weak instead of #ifdef (Dave Hansen)
- Only have start gap on shadow stack (Andy Luto)
- Create stack_guard_start_gap() to not duplicate code
in an arch version of vm_start_gap() (Dave Hansen)
- Improve commit log partly with verbiage from (Dave Hansen)
Yu-cheng v25:
- Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c.
---
include/linux/mm.h | 31 ++++++++++++++++++++++++++-----
1 file changed, 26 insertions(+), 5 deletions(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:20:53
From: Yu-cheng Yu <redacted>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
Account shadow stack pages to stack memory.
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v3:
- Remove unneeded VM_SHADOW_STACK check in accountable_mapping()
(Kirill)
v2:
- Remove is_shadow_stack_mapping() and just change it to directly bitwise
and VM_SHADOW_STACK.
Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.
Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping().
---
mm/mmap.c | 2 ++
1 file changed, 2 insertions(+)
@@ -1042,6 +1042,7 @@ unsigned long do_mmap(struct file *file,unsignedlonglen,unsignedlongprot,unsignedlongflags,+vm_flags_tvm_flags,unsignedlongpgoff,unsignedlong*populate,structlist_head*uf)
@@ -1049,7 +1050,6 @@ unsigned long do_mmap(struct file *file,structvm_area_struct*vma;structvm_region*region;structrb_node*rb;-vm_flags_tvm_flags;unsignedlongcapabilities,result;intret;MA_STATE(mas,¤t->mm->mm_mt,0,0);
@@ -1069,7 +1069,7 @@ unsigned long do_mmap(struct file *file,/* we've determined that we can make the mapping, now translate what we*nowknowintoVMAflags*/-vm_flags=determine_vm_flags(file,prot,flags,capabilities);+vm_flags|=determine_vm_flags(file,prot,flags,capabilities);/* we're going to need to record the mapping */
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:21:11
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
Shadow stack memory is writable only in very specific, controlled ways.
However, since it is writable, the kernel treats it as such. As a result
there remain many ways for userspace to trigger the kernel to write to
shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a
little less exposed, block writable GUPs for shadow stack VMAs.
Still allow FOLL_FORCE to write through shadow stack protections, as it
does for read-only protections.
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v3:
- Add comment in __pte_access_permitted() (Dave)
- Remove unneeded shadow stack specific check in
__pte_access_permitted() (Jann)
---
arch/x86/include/asm/pgtable.h | 5 +++++
mm/gup.c | 2 +-
2 files changed, 6 insertions(+), 1 deletion(-)
@@ -982,7 +982,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)return-EFAULT;if(write){-if(!(vm_flags&VM_WRITE)){+if(!(vm_flags&VM_WRITE)||(vm_flags&VM_SHADOW_STACK)){if(!(gup_flags&FOLL_FORCE))return-EFAULT;/* hugetlb does not support FOLL_FORCE|FOLL_WRITE. */
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:21:24
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which require some core mm changes to function
properly.
One of the properties is that the shadow stack pointer (SSP), which is a
CPU register that points to the shadow stack like the stack pointer points
to the stack, can't be pointing outside of the 32 bit address space when
the CPU is executing in 32 bit mode. It is desirable to prevent executing
in 32 bit mode when shadow stack is enabled because the kernel can't easily
support 32 bit signals.
On x86 it is possible to transition to 32 bit mode without any special
interaction with the kernel, by doing a "far call" to a 32 bit segment.
So the shadow stack implementation can use this address space behavior
as a feature, by enforcing that shadow stack memory is always crated
outside of the 32 bit address space. This way userspace will trigger a
general protection fault which will in turn trigger a segfault if it
tries to transition to 32 bit mode with shadow stack enabled.
This provides a clean error generating border for the user if they try
attempt to do 32 bit mode shadow stack, rather than leave the kernel in a
half working state for userspace to be surprised by.
So to allow future shadow stack enabling patches to map shadow stacks
out of the 32 bit address space, introduce MAP_ABOVE4G. The behavior
is pretty much like MAP_32BIT, except that it has the opposite address
range. The are a few differences though.
If both MAP_32BIT and MAP_ABOVE4G are provided, the kernel will use the
MAP_ABOVE4G behavior. Like MAP_32BIT, MAP_ABOVE4G is ignored in a 32 bit
syscall.
Since the default search behavior is top down, the normal kaslr base can
be used for MAP_ABOVE4G. This is unlike MAP_32BIT which has to add it's
own randomization in the bottom up case.
For MAP_32BIT, only the bottom up search path is used. For MAP_ABOVE4G
both are potentially valid, so both are used. In the bottomup search
path, the default behavior is already consistent with MAP_ABOVE4G since
mmap base should be above 4GB.
Without MAP_ABOVE4G, the shadow stack will already normally be above 4GB.
So without introducing MAP_ABOVE4G, trying to transition to 32 bit mode
with shadow stack enabled would usually segfault anyway. This is already
pretty decent guard rails. But the addition of MAP_ABOVE4G is some small
complexity spent to make it make it more complete.
Tested-by: Pengfei Xu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v5:
- New patch
---
arch/x86/include/uapi/asm/mman.h | 1 +
arch/x86/kernel/sys_x86_64.c | 6 +++++-
include/linux/mman.h | 4 ++++
3 files changed, 10 insertions(+), 1 deletion(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:21:29
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One sharp edge is that PTEs that are both Write=0 and Dirty=1 are
treated as shadow by the CPU, but this combination used to be created by
the kernel on x86. Previous patches have changed the kernel to now avoid
creating these PTEs unless they are for shadow stack memory. In case any
missed corners of the kernel are still creating PTEs like this for
non-shadow stack memory, and to catch any re-introductions of the logic,
warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow
stack VMAs when they are being zapped. This won't catch transient cases
but should have decent coverage. It will be compiled out when shadow
stack is not configured.
In order to check if a pte is shadow stack in core mm code, add two arch
breakouts arch_check_zapped_pte/pmd(). This will allow shadow stack
specific code to be kept in arch/x86.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Add arch breakout to remove shstk from core MM code.
v5:
- Fix typo in commit log
v3:
- New patch
---
arch/x86/include/asm/pgtable.h | 6 ++++++
arch/x86/mm/pgtable.c | 12 ++++++++++++
include/linux/pgtable.h | 14 ++++++++++++++
mm/huge_memory.c | 1 +
mm/memory.c | 1 +
5 files changed, 34 insertions(+)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:21:40
When user shadow stack is use, Write=0,Dirty=1 is treated by the CPU as
shadow stack memory. So for shadow stack memory this bit combination is
valid, but when Dirty=1,Write=1 (conventionally writable) memory is being
write protected, the kernel has been taught to transition the Dirty=1
bit to SavedDirty=1, to avoid inadvertently creating shadow stack
memory. It does this inside pte_wrprotect() because it knows the PTE is
not intended to be a writable shadow stack entry, it is supposed to be
write protected.
However, when a PTE is created by a raw prot using mk_pte(), mk_pte()
can't know whether to adjust Dirty=1 to SavedDirty=1. It can't
distinguish between the caller intending to create a shadow stack PTE or
needing the SavedDirty shift.
The kernel has been updated to not do this, and so Write=0,Dirty=1
memory should only be created by the pte_mkfoo() helpers. Add a warning
to make sure no new mk_pte() start doing this.
Tested-by: Pengfei Xu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- New patch (Note, this has already been a useful warning, it caught the
newly added set_memory_rox() doing this)
---
arch/x86/include/asm/pgtable.h | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:21:44
From: "Kirill A. Shutemov" <redacted>
Add three new arch_prctl() handles:
- ARCH_SHSTK_ENABLE/DISABLE enables or disables the specified
feature. Returns 0 on success or an error.
- ARCH_SHSTK_LOCK prevents future disabling or enabling of the
specified feature. Returns 0 on success or an error
The features are handled per-thread and inherited over fork(2)/clone(2),
but reset on exec().
This is preparation patch. It does not implement any features.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Kirill A. Shutemov <redacted>
[tweaked with feedback from tglx]
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v4:
- Remove references to CET and replace with shadow stack (Peterz)
v3:
- Move shstk.c Makefile changes earlier (Kees)
- Add #ifdef around features_locked and features (Kees)
- Encapsulate features reset earlier in reset_thread_features() so
features and features_locked are not referenced in code that would be
compiled !CONFIG_X86_USER_SHADOW_STACK. (Kees)
- Fix typo in commit log (Kees)
- Switch arch_prctl() numbers to avoid conflict with LAM
v2:
- Only allow one enable/disable per call (tglx)
- Return error code like a normal arch_prctl() (Alexander Potapenko)
- Make CET only (tglx)
---
arch/x86/include/asm/processor.h | 6 +++++
arch/x86/include/asm/shstk.h | 21 +++++++++++++++
arch/x86/include/uapi/asm/prctl.h | 6 +++++
arch/x86/kernel/Makefile | 2 ++
arch/x86/kernel/process_64.c | 7 ++++-
arch/x86/kernel/shstk.c | 44 +++++++++++++++++++++++++++++++
6 files changed, 85 insertions(+), 1 deletion(-)
create mode 100644 arch/x86/include/asm/shstk.h
create mode 100644 arch/x86/kernel/shstk.c
@@ -475,6 +476,11 @@ struct thread_struct {*/u32pkru;+#ifdef CONFIG_X86_USER_SHADOW_STACK+unsignedlongfeatures;+unsignedlongfeatures_locked;+#endif+/* Floating point and extended processor state */structfpufpu;/*
@@ -514,6 +514,8 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip,load_gs_index(__USER_DS);}+reset_thread_features();+loadsegment(fs,0);loadsegment(es,_ds);loadsegment(ds,_ds);
@@ -830,7 +832,10 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)caseARCH_MAP_VDSO_64:returnprctl_map_vdso(&vdso_image_64,arg2);#endif-+caseARCH_SHSTK_ENABLE:+caseARCH_SHSTK_DISABLE:+caseARCH_SHSTK_LOCK:+returnshstk_prctl(task,option,arg2);default:ret=-EINVAL;break;
@@ -0,0 +1,44 @@+// SPDX-License-Identifier: GPL-2.0+/*+*shstk.c-Intelshadowstacksupport+*+*Copyright(c)2021,IntelCorporation.+*Yu-chengYu<yu-cheng.yu@intel.com>+*/++#include<linux/sched.h>+#include<linux/bitops.h>+#include<asm/prctl.h>++voidreset_thread_features(void)+{+current->thread.features=0;+current->thread.features_locked=0;+}++longshstk_prctl(structtask_struct*task,intoption,unsignedlongfeatures)+{+if(option==ARCH_SHSTK_LOCK){+task->thread.features_locked|=features;+return0;+}++/* Don't allow via ptrace */+if(task!=current)+return-EINVAL;++/* Do not allow to change locked features */+if(features&task->thread.features_locked)+return-EPERM;++/* Only support enabling/disabling one feature at a time. */+if(hweight_long(features)>1)+return-EINVAL;++if(option==ARCH_SHSTK_DISABLE){+return-EINVAL;+}++/* Handle ARCH_SHSTK_ENABLE */+return-EINVAL;+}
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:21:48
From: Yu-cheng Yu <redacted>
Introduce basic shadow stack enabling/disabling/allocation routines.
A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag
and has a fixed size of min(RLIMIT_STACK, 4GB).
Keep the task's shadow stack address and size in thread_struct. This will
be copied when cloning new threads, but needs to be cleared during exec,
so add a function to do this.
Do not support IA32 emulation or x32.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v5:
- Switch to EOPNOTSUPP
- Use MAP_ABOVE4G
- Move set_clr_bits_msrl() to patch where it is first used
v4:
- Just set MSR_IA32_U_CET when disabling shadow stack, since we don't
have IBT yet. (Peterz)
v3:
- Use define for set_clr_bits_msrl() (Kees)
- Make some functions static (Kees)
- Change feature_foo() to features_foo() (Kees)
- Centralize shadow stack size rlimit checks (Kees)
- Disable x32 support
v2:
- Get rid of unnecessary shstk->base checks
- Don't support IA32 emulation
---
arch/x86/include/asm/processor.h | 2 +
arch/x86/include/asm/shstk.h | 7 ++
arch/x86/include/uapi/asm/prctl.h | 3 +
arch/x86/kernel/shstk.c | 145 ++++++++++++++++++++++++++++++
4 files changed, 157 insertions(+)
@@ -479,6 +479,8 @@ struct thread_struct {#ifdef CONFIG_X86_USER_SHADOW_STACKunsignedlongfeatures;unsignedlongfeatures_locked;++structthread_shstkshstk;#endif/* Floating point and extended processor state */
@@ -8,14 +8,159 @@#include<linux/sched.h>#include<linux/bitops.h>+#include<linux/types.h>+#include<linux/mm.h>+#include<linux/mman.h>+#include<linux/slab.h>+#include<linux/uaccess.h>+#include<linux/sched/signal.h>+#include<linux/compat.h>+#include<linux/sizes.h>+#include<linux/user.h>+#include<asm/msr.h>+#include<asm/fpu/xstate.h>+#include<asm/fpu/types.h>+#include<asm/shstk.h>+#include<asm/special_insns.h>+#include<asm/fpu/api.h>#include<asm/prctl.h>+staticboolfeatures_enabled(unsignedlongfeatures)+{+returncurrent->thread.features&features;+}++staticvoidfeatures_set(unsignedlongfeatures)+{+current->thread.features|=features;+}++staticvoidfeatures_clr(unsignedlongfeatures)+{+current->thread.features&=~features;+}++staticunsignedlongalloc_shstk(unsignedlongsize)+{+intflags=MAP_ANONYMOUS|MAP_PRIVATE|MAP_ABOVE4G;+structmm_struct*mm=current->mm;+unsignedlongaddr,unused;++mmap_write_lock(mm);+addr=do_mmap(NULL,addr,size,PROT_READ,flags,+VM_SHADOW_STACK|VM_WRITE,0,&unused,NULL);++mmap_write_unlock(mm);++returnaddr;+}++staticunsignedlongadjust_shstk_size(unsignedlongsize)+{+if(size)+returnPAGE_ALIGN(size);++returnPAGE_ALIGN(min_t(unsignedlonglong,rlimit(RLIMIT_STACK),SZ_4G));+}++staticvoidunmap_shadow_stack(u64base,u64size)+{+while(1){+intr;++r=vm_munmap(base,size);++/*+*vm_munmap()returns-EINTRwhenmmap_lockisheldby+*somethingelse,andthatlockshouldnotbeheldfora+*longtime.Retryitforthecase.+*/+if(r==-EINTR){+cond_resched();+continue;+}++/*+*Forallothertypesofvm_munmap()failure,eitherthe+*systemisoutofmemoryorthereisbug.+*/+WARN_ON_ONCE(r);+break;+}+}++staticintshstk_setup(void)+{+structthread_shstk*shstk=¤t->thread.shstk;+unsignedlongaddr,size;++/* Already enabled */+if(features_enabled(ARCH_SHSTK_SHSTK))+return0;++/* Also not supported for 32 bit and x32 */+if(!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)||in_32bit_syscall())+return-EOPNOTSUPP;++size=adjust_shstk_size(0);+addr=alloc_shstk(size);+if(IS_ERR_VALUE(addr))+returnPTR_ERR((void*)addr);++fpregs_lock_and_load();+wrmsrl(MSR_IA32_PL3_SSP,addr+size);+wrmsrl(MSR_IA32_U_CET,CET_SHSTK_EN);+fpregs_unlock();++shstk->base=addr;+shstk->size=size;+features_set(ARCH_SHSTK_SHSTK);++return0;+}+voidreset_thread_features(void){+memset(¤t->thread.shstk,0,sizeof(structthread_shstk));current->thread.features=0;current->thread.features_locked=0;}+voidshstk_free(structtask_struct*tsk)+{+structthread_shstk*shstk=&tsk->thread.shstk;++if(!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)||+!features_enabled(ARCH_SHSTK_SHSTK))+return;++if(!tsk->mm)+return;++unmap_shadow_stack(shstk->base,shstk->size);+}++staticintshstk_disable(void)+{+if(!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))+return-EOPNOTSUPP;++/* Already disabled? */+if(!features_enabled(ARCH_SHSTK_SHSTK))+return0;++fpregs_lock_and_load();+/* Disable WRSS too when disabling shadow stack */+wrmsrl(MSR_IA32_U_CET,0);+wrmsrl(MSR_IA32_PL3_SSP,0);+fpregs_unlock();++shstk_free(current);+features_clr(ARCH_SHSTK_SHSTK);++return0;+}+longshstk_prctl(structtask_struct*task,intoption,unsignedlongfeatures){if(option==ARCH_SHSTK_LOCK){
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:22:04
From: Yu-cheng Yu <redacted>
When a process is duplicated, but the child shares the address space with
the parent, there is potential for the threads sharing a single stack to
cause conflicts for each other. In the normal non-cet case this is handled
in two ways.
With regular CLONE_VM a new stack is provided by userspace such that the
parent and child have different stacks.
For vfork, the parent is suspended until the child exits. So as long as
the child doesn't return from the vfork()/CLONE_VFORK calling function and
sticks to a limited set of operations, the parent and child can share the
same stack.
For shadow stack, these scenarios present similar sharing problems. For the
CLONE_VM case, the child and the parent must have separate shadow stacks.
Instead of changing clone to take a shadow stack, have the kernel just
allocate one and switch to it.
Use stack_size passed from clone3() syscall for thread shadow stack size. A
compat-mode thread shadow stack size is further reduced to 1/4. This
allows more threads to run in a 32-bit address space. The clone() does not
pass stack_size, which was added to clone3(). In that case, use
RLIMIT_STACK size and cap to 4 GB.
For shadow stack enabled vfork(), the parent and child can share the same
shadow stack, like they can share a normal stack. Since the parent is
suspended until the child terminates, the child will not interfere with
the parent while executing as long as it doesn't return from the vfork()
and overwrite up the shadow stack. The child can safely overwrite down
the shadow stack, as the parent can just overwrite this later. So CET does
not add any additional limitations for vfork().
Userspace implementing posix vfork() can actually prevent the child from
returning from the vfork() calling function, using CET. Glibc does this
by adjusting the shadow stack pointer in the child, so that the child
receives a #CP if it tries to return from vfork() calling function.
Free the shadow stack on thread exit by doing it in mm_release(). Skip
this when exiting a vfork() child since the stack is shared in the
parent.
During this operation, the shadow stack pointer of the new thread needs
to be updated to point to the newly allocated shadow stack. Since the
ability to do this is confined to the FPU subsystem, change
fpu_clone() to take the new shadow stack pointer, and update it
internally inside the FPU subsystem. This part was suggested by Thomas
Gleixner.
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Suggested-by: Thomas Gleixner <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v3:
- Fix update_fpu_shstk() stub (Mike Rapoport)
- Fix chunks around alloc_shstk() in wrong patch (Kees)
- Fix stack_size/flags swap (Kees)
- Use centralized stack size logic (Kees)
v2:
- Have fpu_clone() take new shadow stack pointer and update SSP in
xsave buffer for new task. (tglx)
v1:
- Expand commit log.
- Add more comments.
- Switch to xsave helpers.
Yu-cheng v30:
- Update comments about clone()/clone3(). (Borislav Petkov)
---
arch/x86/include/asm/fpu/sched.h | 3 ++-
arch/x86/include/asm/mmu_context.h | 2 ++
arch/x86/include/asm/shstk.h | 7 +++++
arch/x86/kernel/fpu/core.c | 41 +++++++++++++++++++++++++++-
arch/x86/kernel/process.c | 18 ++++++++++++-
arch/x86/kernel/shstk.c | 43 ++++++++++++++++++++++++++++--
6 files changed, 109 insertions(+), 5 deletions(-)
@@ -552,8 +552,41 @@ static inline void fpu_inherit_perms(struct fpu *dst_fpu)}}+#ifdef CONFIG_X86_USER_SHADOW_STACK+staticintupdate_fpu_shstk(structtask_struct*dst,unsignedlongssp)+{+structcet_user_state*xstate;++/* If ssp update is not needed. */+if(!ssp)+return0;++xstate=get_xsave_addr(&dst->thread.fpu.fpstate->regs.xsave,+XFEATURE_CET_USER);++/*+*Ifthereisanon-zerossp,then'dst'mustbeconfiguredwithashadow+*stackandthefpustateshouldbeuptodatesinceitwasjustcopied+*fromtheparentinfpu_clone().Sotheremustbeavalidnon-initCET+*statelocationinthebuffer.+*/+if(WARN_ON_ONCE(!xstate))+return1;++xstate->user_ssp=(u64)ssp;++return0;+}+#else+staticintupdate_fpu_shstk(structtask_struct*dst,unsignedlongshstk_addr)+{+return0;+}+#endif+/* Clone current's FPU state on fork */-intfpu_clone(structtask_struct*dst,unsignedlongclone_flags,boolminimal)+intfpu_clone(structtask_struct*dst,unsignedlongclone_flags,boolminimal,+unsignedlongssp){structfpu*src_fpu=¤t->thread.fpu;structfpu*dst_fpu=&dst->thread.fpu;
@@ -613,6 +646,12 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal)if(use_xsave())dst_fpu->fpstate->regs.xsave.header.xfeatures&=~XFEATURE_MASK_PASID;+/*+*Updateshadowstackpointer,incaseitchangedduringclone.+*/+if(update_fpu_shstk(dst,ssp))+return1;+trace_x86_fpu_copy_src(src_fpu);trace_x86_fpu_copy_dst(dst_fpu);
@@ -47,7 +47,7 @@ static unsigned long alloc_shstk(unsigned long size)unsignedlongaddr,unused;mmap_write_lock(mm);-addr=do_mmap(NULL,addr,size,PROT_READ,flags,+addr=do_mmap(NULL,0,size,PROT_READ,flags,VM_SHADOW_STACK|VM_WRITE,0,&unused,NULL);mmap_write_unlock(mm);
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:22:19
From: Yu-cheng Yu <redacted>
Shadow stacks are normally written to via CALL/RET or specific CET
instructions like RSTORSSP/SAVEPREVSSP. However during some Linux
operations the kernel will need to write to directly using the ring-0 only
WRUSS instruction.
A shadow stack restore token marks a restore point of the shadow stack, and
the address in a token must point directly above the token, which is within
the same shadow stack. This is distinctively different from other pointers
on the shadow stack, since those pointers point to executable code area.
Introduce token setup and verify routines. Also introduce WRUSS, which is
a kernel-mode instruction but writes directly to user shadow stack.
In future patches that enable shadow stack to work with signals, the kernel
will need something to denote the point in the stack where sigreturn may be
called. This will prevent attackers calling sigreturn at arbitrary places
in the stack, in order to help prevent SROP attacks.
To do this, something that can only be written by the kernel needs to be
placed on the shadow stack. This can be accomplished by setting bit 63 in
the frame written to the shadow stack. Userspace return addresses can't
have this bit set as it is in the kernel range. It is also can't be a
valid restore token.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v5:
- Fix typo in commit log
v3:
- Drop shstk_check_rstor_token()
- Fail put_shstk_data() if bit 63 is set in the data (Kees)
- Add comment in create_rstor_token() (Kees)
- Pull in create_rstor_token() changes from future patch (Kees)
v2:
- Add data helpers for writing to shadow stack.
v1:
- Use xsave helpers.
---
arch/x86/include/asm/special_insns.h | 13 +++++
arch/x86/kernel/shstk.c | 73 ++++++++++++++++++++++++++++
2 files changed, 86 insertions(+)
@@ -40,6 +42,35 @@ static void features_clr(unsigned long features)current->thread.features&=~features;}+/*+*Createarestoretokenontheshadowstack.Atokenisalways8-byte+*andalignedto8.+*/+staticintcreate_rstor_token(unsignedlongssp,unsignedlong*token_addr)+{+unsignedlongaddr;++/* Token must be aligned */+if(!IS_ALIGNED(ssp,8))+return-EINVAL;++addr=ssp-SS_FRAME_SIZE;++/*+*SSPisaligned,soreservedbitsandmodebitareazero,justmark+*thetoken64-bit.+*/+ssp|=BIT(0);++if(write_user_shstk_64((u64__user*)addr,(u64)ssp))+return-EFAULT;++if(token_addr)+*token_addr=addr;++return0;+}+staticunsignedlongalloc_shstk(unsignedlongsize){intflags=MAP_ANONYMOUS|MAP_PRIVATE|MAP_ABOVE4G;
@@ -159,6 +190,48 @@ int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags,return0;}+staticunsignedlongget_user_shstk_addr(void)+{+unsignedlonglongssp;++fpregs_lock_and_load();++rdmsrl(MSR_IA32_PL3_SSP,ssp);++fpregs_unlock();++returnssp;+}++staticintput_shstk_data(u64__user*addr,u64data)+{+if(WARN_ON_ONCE(data&BIT(63)))+return-EINVAL;++/*+*Markthehighbitsothatthesigframecan'tbeprocessedasa+*returnaddress.+*/+if(write_user_shstk_64(addr,data|BIT(63)))+return-EFAULT;+return0;+}++staticintget_shstk_data(unsignedlong*data,unsignedlong__user*addr)+{+unsignedlongldata;++if(unlikely(get_user(ldata,addr)))+return-EFAULT;++if(!(ldata&BIT(63)))+return-EINVAL;++*data=ldata&~BIT(63);++return0;+}+voidshstk_free(structtask_struct*tsk){structthread_shstk*shstk=&tsk->thread.shstk;
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:22:21
From: Yu-cheng Yu <redacted>
When a signal is handled normally the context is pushed to the stack
before handling it. For shadow stacks, since the shadow stack only track's
return addresses, there isn't any state that needs to be pushed. However,
there are still a few things that need to be done. These things are
userspace visible and which will be kernel ABI for shadow stacks.
One is to make sure the restorer address is written to shadow stack, since
the signal handler (if not changing ucontext) returns to the restorer, and
the restorer calls sigreturn. So add the restorer on the shadow stack
before handling the signal, so there is not a conflict when the signal
handler returns to the restorer.
The other thing to do is to place some type of checkable token on the
thread's shadow stack before handling the signal and check it during
sigreturn. This is an extra layer of protection to hamper attackers
calling sigreturn manually as in SROP-like attacks.
For this token we can use the shadow stack data format defined earlier.
Have the data pushed be the previous SSP. In the future the sigreturn
might want to return back to a different stack. Storing the SSP (instead
of a restore offset or something) allows for future functionality that
may want to restore to a different stack.
So, when handling a signal push
- the SSP pointing in the shadow stack data format
- the restorer address below the restore token.
In sigreturn, verify SSP is stored in the data format and pop the shadow
stack.
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Cyrill Gorcunov <redacted>
Cc: Florian Weimer <redacted>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <redacted>
---
v3:
- Drop shstk_setup_rstor_token() (Kees)
- Drop x32 signal support, since x32 support is dropped
v2:
- Switch to new shstk signal format
v1:
- Use xsave helpers.
- Expand commit log.
Yu-cheng v27:
- Eliminate saving shadow stack pointer to signal context.
---
arch/x86/include/asm/shstk.h | 5 ++
arch/x86/kernel/shstk.c | 98 ++++++++++++++++++++++++++++++++++++
arch/x86/kernel/signal.c | 1 +
arch/x86/kernel/signal_64.c | 6 +++
4 files changed, 110 insertions(+)
@@ -232,6 +232,104 @@ static int get_shstk_data(unsigned long *data, unsigned long __user *addr)return0;}+staticintshstk_push_sigframe(unsignedlong*ssp)+{+unsignedlongtarget_ssp=*ssp;++/* Token must be aligned */+if(!IS_ALIGNED(*ssp,8))+return-EINVAL;++if(!IS_ALIGNED(target_ssp,8))+return-EINVAL;++*ssp-=SS_FRAME_SIZE;+if(put_shstk_data((void*__user)*ssp,target_ssp))+return-EFAULT;++return0;+}++staticintshstk_pop_sigframe(unsignedlong*ssp)+{+unsignedlongtoken_addr;+interr;++err=get_shstk_data(&token_addr,(unsignedlong__user*)*ssp);+if(unlikely(err))+returnerr;++/* Restore SSP aligned? */+if(unlikely(!IS_ALIGNED(token_addr,8)))+return-EINVAL;++/* SSP in userspace? */+if(unlikely(token_addr>=TASK_SIZE_MAX))+return-EINVAL;++*ssp=token_addr;++return0;+}++intsetup_signal_shadow_stack(structksignal*ksig)+{+void__user*restorer=ksig->ka.sa.sa_restorer;+unsignedlongssp;+interr;++if(!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)||+!features_enabled(ARCH_SHSTK_SHSTK))+return0;++if(!restorer)+return-EINVAL;++ssp=get_user_shstk_addr();+if(unlikely(!ssp))+return-EINVAL;++err=shstk_push_sigframe(&ssp);+if(unlikely(err))+returnerr;++/* Push restorer address */+ssp-=SS_FRAME_SIZE;+err=write_user_shstk_64((u64__user*)ssp,(u64)restorer);+if(unlikely(err))+return-EFAULT;++fpregs_lock_and_load();+wrmsrl(MSR_IA32_PL3_SSP,ssp);+fpregs_unlock();++return0;+}++intrestore_signal_shadow_stack(void)+{+unsignedlongssp;+interr;++if(!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)||+!features_enabled(ARCH_SHSTK_SHSTK))+return0;++ssp=get_user_shstk_addr();+if(unlikely(!ssp))+return-EINVAL;++err=shstk_pop_sigframe(&ssp);+if(unlikely(err))+returnerr;++fpregs_lock_and_load();+wrmsrl(MSR_IA32_PL3_SSP,ssp);+fpregs_unlock();++return0;+}+voidshstk_free(structtask_struct*tsk){structthread_shstk*shstk=&tsk->thread.shstk;
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:22:24
When operating with shadow stacks enabled, the kernel will automatically
allocate shadow stacks for new threads, however in some cases userspace
will need additional shadow stacks. The main example of this is the
ucontext family of functions, which require userspace allocating and
pivoting to userspace managed stacks.
Unlike most other user memory permissions, shadow stacks need to be
provisioned with special data in order to be useful. They need to be setup
with a restore token so that userspace can pivot to them via the RSTORSSP
instruction. But, the security design of shadow stack's is that they
should not be written to except in limited circumstances. This presents a
problem for userspace, as to how userspace can provision this special
data, without allowing for the shadow stack to be generally writable.
Previously, a new PROT_SHADOW_STACK was attempted, which could be
mprotect()ed from RW permissions after the data was provisioned. This was
found to not be secure enough, as other thread's could write to the
shadow stack during the writable window.
The kernel can use a special instruction, WRUSS, to write directly to
userspace shadow stacks. So the solution can be that memory can be mapped
as shadow stack permissions from the beginning (never generally writable
in userspace), and the kernel itself can write the restore token.
First, a new madvise() flag was explored, which could operate on the
PROT_SHADOW_STACK memory. This had a couple downsides:
1. Extra checks were needed in mprotect() to prevent writable memory from
ever becoming PROT_SHADOW_STACK.
2. Extra checks/vma state were needed in the new madvise() to prevent
restore tokens being written into the middle of pre-used shadow stacks.
It is ideal to prevent restore tokens being added at arbitrary
locations, so the check was to make sure the shadow stack had never been
written to.
3. It stood out from the rest of the madvise flags, as more of direct
action than a hint at future desired behavior.
So rather than repurpose two existing syscalls (mmap, madvise) that don't
quite fit, just implement a new map_shadow_stack syscall to allow
userspace to map and setup new shadow stacks in one step. While ucontext
is the primary motivator, userspace may have other unforeseen reasons to
setup it's own shadow stacks using the WRSS instruction. Towards this
provide a flag so that stacks can be optionally setup securely for the
common case of ucontext without enabling WRSS. Or potentially have the
kernel set up the shadow stack in some new way.
The following example demonstrates how to create a new shadow stack with
map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v5:
- Fix addr/mapped_addr (Kees)
- Switch to EOPNOTSUPP (Kees suggested ENOTSUPP, but checkpatch
suggests this)
- Return error for addresses below 4G
v3:
- Change syscall common -> 64 (Kees)
- Use bit shift notation instead of 0x1 for uapi header (Kees)
- Call do_mmap() with MAP_FIXED_NOREPLACE (Kees)
- Block unsupported flags (Kees)
- Require size >= 8 to set token (Kees)
v2:
- Change syscall to take address like mmap() for CRIU's usage
v1:
- New patch (replaces PROT_SHADOW_STACK).
---
arch/x86/entry/syscalls/syscall_64.tbl | 1 +
arch/x86/include/uapi/asm/mman.h | 3 ++
arch/x86/kernel/shstk.c | 59 ++++++++++++++++++++++----
include/linux/syscalls.h | 1 +
include/uapi/asm-generic/unistd.h | 2 +-
kernel/sys_ni.c | 1 +
6 files changed, 58 insertions(+), 9 deletions(-)
@@ -372,6 +372,7 @@ 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv 450 common set_mempolicy_home_node sys_set_mempolicy_home_node+451 64 map_shadow_stack sys_map_shadow_stack # # Due to a historical design error, certain syscalls are numbered differently
@@ -13,6 +13,9 @@((key)&0x8?VM_PKEY_BIT3:0))#endif+/* Flags for map_shadow_stack(2) */+#define SHADOW_STACK_SET_TOKEN (1ULL << 0) /* Set up a restore token in the shadow stack */+#include<asm-generic/mman.h>#endif /* _ASM_X86_MMAN_H */
@@ -71,19 +72,31 @@ static int create_rstor_token(unsigned long ssp, unsigned long *token_addr)return0;}-staticunsignedlongalloc_shstk(unsignedlongsize)+staticunsignedlongalloc_shstk(unsignedlongaddr,unsignedlongsize,+unsignedlongtoken_offset,boolset_res_tok){intflags=MAP_ANONYMOUS|MAP_PRIVATE|MAP_ABOVE4G;structmm_struct*mm=current->mm;-unsignedlongaddr,unused;+unsignedlongmapped_addr,unused;-mmap_write_lock(mm);-addr=do_mmap(NULL,0,size,PROT_READ,flags,-VM_SHADOW_STACK|VM_WRITE,0,&unused,NULL);+if(addr)+flags|=MAP_FIXED_NOREPLACE;+mmap_write_lock(mm);+mapped_addr=do_mmap(NULL,addr,size,PROT_READ,flags,+VM_SHADOW_STACK|VM_WRITE,0,&unused,NULL);mmap_write_unlock(mm);-returnaddr;+if(!set_res_tok||IS_ERR_VALUE(mapped_addr))+gotoout;++if(create_rstor_token(mapped_addr+token_offset,NULL)){+vm_munmap(mapped_addr,size);+return-EINVAL;+}++out:+returnmapped_addr;}staticunsignedlongadjust_shstk_size(unsignedlongsize)
@@ -134,7 +147,7 @@ static int shstk_setup(void)return-EOPNOTSUPP;size=adjust_shstk_size(0);-addr=alloc_shstk(size);+addr=alloc_shstk(0,size,0,false);if(IS_ERR_VALUE(addr))returnPTR_ERR((void*)addr);
@@ -178,7 +191,7 @@ int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags,return0;size=adjust_shstk_size(stack_size);-addr=alloc_shstk(size);+addr=alloc_shstk(0,size,0,false);if(IS_ERR_VALUE(addr))returnPTR_ERR((void*)addr);
@@ -371,6 +384,36 @@ static int shstk_disable(void)return0;}+SYSCALL_DEFINE3(map_shadow_stack,unsignedlong,addr,unsignedlong,size,unsignedint,flags)+{+boolset_tok=flags&SHADOW_STACK_SET_TOKEN;+unsignedlongaligned_size;++if(!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))+return-EOPNOTSUPP;++if(flags&~SHADOW_STACK_SET_TOKEN)+return-EINVAL;++/* If there isn't space for a token */+if(set_tok&&size<8)+return-EINVAL;++if(addr&&addr<=0xFFFFFFFF)+return-EINVAL;++/*+*Anoverflowwouldresultinattemptingtowritetherestoretoken+*tothewronglocation.Notcatastrophic,butjustreturntheright+*errorcodeandblockit.+*/+aligned_size=PAGE_ALIGN(size);+if(aligned_size<size)+return-EOVERFLOW;++returnalloc_shstk(addr,aligned_size,size,set_tok);+}+longshstk_prctl(structtask_struct*task,intoption,unsignedlongfeatures){if(option==ARCH_SHSTK_LOCK){
@@ -1058,6 +1058,7 @@ asmlinkage long sys_memfd_secret(unsigned int flags);asmlinkagelongsys_set_mempolicy_home_node(unsignedlongstart,unsignedlonglen,unsignedlonghome_node,unsignedlongflags);+asmlinkagelongsys_map_shadow_stack(unsignedlongaddr,unsignedlongsize,unsignedintflags);/**Architecture-specificsystemcalls
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:22:40
For the current shadow stack implementation, shadow stacks contents can't
easily be provisioned with arbitrary data. This property helps apps
protect themselves better, but also restricts any potential apps that may
want to do exotic things at the expense of a little security.
The x86 shadow stack feature introduces a new instruction, WRSS, which
can be enabled to write directly to shadow stack permissioned memory from
userspace. Allow it to get enabled via the prctl interface.
Only enable the userspace WRSS instruction, which allows writes to
userspace shadow stacks from userspace. Do not allow it to be enabled
independently of shadow stack, as HW does not support using WRSS when
shadow stack is disabled.
From a fault handler perspective, WRSS will behave very similar to WRUSS,
which is treated like a user access from a #PF err code perspective.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Make set_clr_bits_msrl() avoid side affects in 'msr'
v5:
- Switch to EOPNOTSUPP
- Move set_clr_bits_msrl() to patch where it is first used
- Commit log formatting
v3:
- Make wrss_control() static
- Fix verbiage in commit log (Kees)
v2:
- Add some commit log verbiage from (Dave Hansen)
v1:
- New patch.
---
arch/x86/include/asm/msr.h | 11 +++++++++++
arch/x86/include/uapi/asm/prctl.h | 1 +
arch/x86/kernel/shstk.c | 32 ++++++++++++++++++++++++++++++-
3 files changed, 43 insertions(+), 1 deletion(-)
@@ -310,6 +310,17 @@ void msrs_free(struct msr *msrs);intmsr_set_bit(u32msr,u8bit);intmsr_clear_bit(u32msr,u8bit);+/* Helper that can never get accidentally un-inlined. */+#define set_clr_bits_msrl(msr, set, clear) do { \+u64__val,__new_val,__msr=msr;\+\+rdmsrl(__msr,__val);\+__new_val=(__val&~(clear))|(set);\+\+if(__new_val!=__val)\+wrmsrl(__msr,__new_val);\+}while(0)+#ifdef CONFIG_SMPintrdmsr_on_cpu(unsignedintcpu,u32msr_no,u32*l,u32*h);intwrmsr_on_cpu(unsignedintcpu,u32msr_no,u32l,u32h);
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:22:43
Applications and loaders can have logic to decide whether to enable
shadow stack. They usually don't report whether shadow stack has been
enabled or not, so there is no way to verify whether an application
actually is protected by shadow stack.
Add two lines in /proc/$PID/status to report enabled and locked features.
Since, this involves referring to arch specific defines in asm/prctl.h,
implement an arch breakout to emit the feature lines.
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Kirill A. Shutemov <redacted>
[Switched to CET, added to commit log]
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v4:
- Remove "CET" references
v3:
- Move to /proc/pid/status (Kees)
v2:
- New patch
---
arch/x86/kernel/cpu/proc.c | 23 +++++++++++++++++++++++
fs/proc/array.c | 6 ++++++
include/linux/proc_fs.h | 2 ++
3 files changed, 31 insertions(+)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:22:49
The kernel now has the main shadow stack functionality to support
applications. Wire in the WRSS and shadow stack enable/disable functions
into the existing shadow stack API skeleton.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v4:
- Remove "CET" references
v2:
- Split from other patches
---
arch/x86/kernel/shstk.c | 8 ++++++++
1 file changed, 8 insertions(+)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:23:10
If an xfeature is saved in a buffer, the xfeature's bit will be set in
xsave->header.xfeatures. The CPU may opt to not save the xfeature if it
is in it's init state. In this case the xfeature buffer address cannot
be retrieved with get_xsave_addr().
Future patches will need to handle the case of writing to an xfeature
that may not be saved. So provide helpers to init an xfeature in an
xsave buffer.
This could of course be done directly by reaching into the xsave buffer,
however this would not be robust against future changes to optimize the
xsave buffer by compacting it. In that case the xsave buffer would need
to be re-arranged as well. So the logic properly belongs encapsulated
in a helper where the logic can be unified.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v2:
- New patch
---
arch/x86/kernel/fpu/xstate.c | 58 +++++++++++++++++++++++++++++-------
arch/x86/kernel/fpu/xstate.h | 6 ++++
2 files changed, 53 insertions(+), 11 deletions(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:23:12
Add a simple selftest for exercising some shadow stack behavior:
- map_shadow_stack syscall and pivot
- Faulting in shadow stack memory
- Handling shadow stack violations
- GUP of shadow stack memory
- mprotect() of shadow stack memory
- Userfaultfd on shadow stack memory
Since this test exercises a recently added syscall manually, it needs
to find the automatically created __NR_foo defines. Per the selftest
documentation, KHDR_INCLUDES can be used to help the selftest Makefile's
find the headers from the kernel source. This way the new selftest can
be built inside the kernel source tree without installing the headers
to the system. So also add KHDR_INCLUDES as described in the selftest
docs, to facilitate this.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Co-developed-by: Yu-cheng Yu <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Tweak mprotect test
- Code style tweaks
v5:
- Update 32 bit signal test with new ABI and better asm
v4:
- Add test for 32 bit signal ABI blocking
v3:
- Change "+m" to "=m" in write_shstk() (Andrew Cooper)
- Fix userfaultfd test with transparent huge pages by doing a
MADV_DONTNEED, since the token write faults in the while stack with
huge pages.
---
tools/testing/selftests/x86/Makefile | 4 +-
.../testing/selftests/x86/test_shadow_stack.c | 676 ++++++++++++++++++
2 files changed, 678 insertions(+), 2 deletions(-)
create mode 100644 tools/testing/selftests/x86/test_shadow_stack.c
@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \test_FCMOVtest_FCOMItest_FISTTP\vdso_restorerTARGETS_C_64BIT_ONLY:=fsgsbasesysret_ripsyscall_numbering\-corrupt_xstate_headeramx+corrupt_xstate_headeramxtest_shadow_stack# Some selftests require 32bit support enabled also on 64bit systemsTARGETS_C_32BIT_NEEDED:=ldt_gdtptrace_syscall
@@ -0,0 +1,676 @@+// SPDX-License-Identifier: GPL-2.0+/*+*Thisprogramtest'sbasickernelshadowstacksupport.Itenablesshadow+*stackmanualviathearch_prctl(),insteadofrelyingonglibc.It's+*Makefiledoesn'tcompilewithshadowstacksupport,soitdoesn'trelyon+*anyparticularglibc.Asaresultitcan'tdoanyoperationsthatrequire+*specialglibcshadowstacksupport(longjmp(),swapcontext(),etc).Just+*sticktothebasicsandhopethecompilerdoesn'tdoanythingstrange.+*/++#define _GNU_SOURCE++#include<sys/syscall.h>+#include<asm/mman.h>+#include<sys/mman.h>+#include<sys/stat.h>+#include<sys/wait.h>+#include<stdio.h>+#include<stdlib.h>+#include<fcntl.h>+#include<unistd.h>+#include<string.h>+#include<errno.h>+#include<stdbool.h>+#include<x86intrin.h>+#include<asm/prctl.h>+#include<sys/prctl.h>+#include<stdint.h>+#include<signal.h>+#include<pthread.h>+#include<sys/ioctl.h>+#include<linux/userfaultfd.h>+#include<setjmp.h>++#define SS_SIZE 0x200000++#if (__GNUC__ < 8) || (__GNUC__ == 8 && __GNUC_MINOR__ < 5)+intmain(intargc,char*argv[])+{+printf("[SKIP]\tCompiler does not support CET.\n");+return0;+}+#else+voidwrite_shstk(unsignedlong*addr,unsignedlongval)+{+asmvolatile("wrssq %[val], (%[addr])\n"+:"=m"(addr)+:[addr]"r"(addr),[val]"r"(val));+}++staticinlineunsignedlong__attribute__((always_inline))get_ssp(void)+{+unsignedlongret=0;++asmvolatile("xor %0, %0; rdsspq %0":"=r"(ret));+returnret;+}++/*+*Foruseininlineenablementofshadowstack.+*+*Theprogramcan'treturnfromthepointwhereshadowstackgetsenabled+*becausetherewillbenoaddressontheshadowstack.Soitcan'tuse+*syscall()forenablement,sinceitisafunction.+*+*Basedoncodefromnolibc.h.Keepacopyherebecausethiscan'tpullinall+*ofnolibc.h.+*/+#define ARCH_PRCTL(arg1, arg2) \+({\+long_ret;\+registerlong_numasm("eax")=__NR_arch_prctl;\+registerlong_arg1asm("rdi")=(long)(arg1);\+registerlong_arg2asm("rsi")=(long)(arg2);\+\+asmvolatile(\+"syscall\n"\+:"=a"(_ret)\+:"r"(_arg1),"r"(_arg2),\+"0"(_num)\+:"rcx","r11","memory","cc"\+);\+_ret;\+})++void*create_shstk(void*addr)+{+return(void*)syscall(__NR_map_shadow_stack,addr,SS_SIZE,SHADOW_STACK_SET_TOKEN);+}++void*create_normal_mem(void*addr)+{+returnmmap(addr,SS_SIZE,PROT_READ|PROT_WRITE,+MAP_PRIVATE|MAP_ANONYMOUS,0,0);+}++voidfree_shstk(void*shstk)+{+munmap(shstk,SS_SIZE);+}++intreset_shstk(void*shstk)+{+returnmadvise(shstk,SS_SIZE,MADV_DONTNEED);+}++voidtry_shstk(unsignedlongnew_ssp)+{+unsignedlongssp;++printf("[INFO]\tnew_ssp = %lx, *new_ssp = %lx\n",+new_ssp,*((unsignedlong*)new_ssp));++ssp=get_ssp();+printf("[INFO]\tchanging ssp from %lx to %lx\n",ssp,new_ssp);++asmvolatile("rstorssp (%0)\n"::"r"(new_ssp));+asmvolatile("saveprevssp");+printf("[INFO]\tssp is now %lx\n",get_ssp());++/* Switch back to original shadow stack */+ssp-=8;+asmvolatile("rstorssp (%0)\n"::"r"(ssp));+asmvolatile("saveprevssp");+}++inttest_shstk_pivot(void)+{+void*shstk=create_shstk(0);++if(shstk==MAP_FAILED){+printf("[FAIL]\tError creating shadow stack: %d\n",errno);+return1;+}+try_shstk((unsignedlong)shstk+SS_SIZE-8);+free_shstk(shstk);++printf("[OK]\tShadow stack pivot\n");+return0;+}++inttest_shstk_faults(void)+{+unsignedlong*shstk=create_shstk(0);++/* Read shadow stack, test if it's zero to not get read optimized out */+if(*shstk!=0)+gotoerr;++/* Wrss memory that was already read. */+write_shstk(shstk,1);+if(*shstk!=1)+gotoerr;++/* Page out memory, so we can wrss it again. */+if(reset_shstk((void*)shstk))+gotoerr;++write_shstk(shstk,1);+if(*shstk!=1)+gotoerr;++printf("[OK]\tShadow stack faults\n");+return0;++err:+return1;+}++unsignedlongsaved_ssp;+unsignedlongsaved_ssp_val;+volatileboolsegv_triggered;++void__attribute__((noinline))violate_ss(void)+{+saved_ssp=get_ssp();+saved_ssp_val=*(unsignedlong*)saved_ssp;++/* Corrupt shadow stack */+printf("[INFO]\tCorrupting shadow stack\n");+write_shstk((void*)saved_ssp,0);+}++voidsegv_handler(intsignum,siginfo_t*si,void*uc)+{+printf("[INFO]\tGenerated shadow stack violation successfully\n");++segv_triggered=true;++/* Fix shadow stack */+write_shstk((void*)saved_ssp,saved_ssp_val);+}++inttest_shstk_violation(void)+{+structsigactionsa;++sa.sa_sigaction=segv_handler;+if(sigaction(SIGSEGV,&sa,NULL))+return1;+sa.sa_flags=SA_SIGINFO;++segv_triggered=false;++/* Make sure segv_triggered is set before violate_ss() */+asmvolatile("":::"memory");++violate_ss();++signal(SIGSEGV,SIG_DFL);++printf("[OK]\tShadow stack violation test\n");++return!segv_triggered;+}++/* Gup test state */+#define MAGIC_VAL 0x12345678+boolis_shstk_access;+void*shstk_ptr;+intfd;++voidreset_test_shstk(void*addr)+{+if(shstk_ptr)+free_shstk(shstk_ptr);+shstk_ptr=create_shstk(addr);+}++voidtest_access_fix_handler(intsignum,siginfo_t*si,void*uc)+{+printf("[INFO]\tViolation from %s\n",is_shstk_access?"shstk access":"normal write");++segv_triggered=true;++/* Fix shadow stack */+if(is_shstk_access){+reset_test_shstk(shstk_ptr);+return;+}++free_shstk(shstk_ptr);+create_normal_mem(shstk_ptr);+}++booltest_shstk_access(void*ptr)+{+is_shstk_access=true;+segv_triggered=false;+write_shstk(ptr,MAGIC_VAL);++asmvolatile("":::"memory");++returnsegv_triggered;+}++booltest_write_access(void*ptr)+{+is_shstk_access=false;+segv_triggered=false;+*(unsignedlong*)ptr=MAGIC_VAL;++asmvolatile("":::"memory");++returnsegv_triggered;+}++boolgup_write(void*ptr)+{+unsignedlongval;++lseek(fd,(unsignedlong)ptr,SEEK_SET);+if(write(fd,&val,sizeof(val))<0)+return1;++return0;+}++boolgup_read(void*ptr)+{+unsignedlongval;++lseek(fd,(unsignedlong)ptr,SEEK_SET);+if(read(fd,&val,sizeof(val))<0)+return1;++return0;+}++inttest_gup(void)+{+structsigactionsa;+intstatus;+pid_tpid;++sa.sa_sigaction=test_access_fix_handler;+if(sigaction(SIGSEGV,&sa,NULL))+return1;+sa.sa_flags=SA_SIGINFO;++segv_triggered=false;++fd=open("/proc/self/mem",O_RDWR);+if(fd==-1)+return1;++reset_test_shstk(0);+if(gup_read(shstk_ptr))+return1;+if(test_shstk_access(shstk_ptr))+return1;+printf("[INFO]\tGup read -> shstk access success\n");++reset_test_shstk(0);+if(gup_write(shstk_ptr))+return1;+if(test_shstk_access(shstk_ptr))+return1;+printf("[INFO]\tGup write -> shstk access success\n");++reset_test_shstk(0);+if(gup_read(shstk_ptr))+return1;+if(!test_write_access(shstk_ptr))+return1;+printf("[INFO]\tGup read -> write access success\n");++reset_test_shstk(0);+if(gup_write(shstk_ptr))+return1;+if(!test_write_access(shstk_ptr))+return1;+printf("[INFO]\tGup write -> write access success\n");++close(fd);++/* COW/gup test */+reset_test_shstk(0);+pid=fork();+if(!pid){+fd=open("/proc/self/mem",O_RDWR);+if(fd==-1)+exit(1);++if(gup_write(shstk_ptr)){+close(fd);+exit(1);+}+close(fd);+exit(0);+}+waitpid(pid,&status,0);+if(WEXITSTATUS(status)){+printf("[FAIL]\tWrite in child failed\n");+return1;+}+if(*(unsignedlong*)shstk_ptr==MAGIC_VAL){+printf("[FAIL]\tWrite in child wrote through to shared memory\n");+return1;+}++printf("[INFO]\tCow gup write -> write access success\n");++free_shstk(shstk_ptr);++signal(SIGSEGV,SIG_DFL);++printf("[OK]\tShadow gup test\n");++return0;+}++inttest_mprotect(void)+{+structsigactionsa;++sa.sa_sigaction=test_access_fix_handler;+if(sigaction(SIGSEGV,&sa,NULL))+return1;+sa.sa_flags=SA_SIGINFO;++segv_triggered=false;++/* mprotect a shadow stack as read only */+reset_test_shstk(0);+if(mprotect(shstk_ptr,SS_SIZE,PROT_READ)<0){+printf("[FAIL]\tmprotect(PROT_READ) failed\n");+return1;+}++/* try to wrss it and fail */+if(!test_shstk_access(shstk_ptr)){+printf("[FAIL]\tShadow stack access to read-only memory succeeded\n");+return1;+}++/*+*Theshadowstackwasresetabovetoresolvethefault,makethenewone+*read-only.+*/+if(mprotect(shstk_ptr,SS_SIZE,PROT_READ)<0){+printf("[FAIL]\tmprotect(PROT_READ) failed\n");+return1;+}++/* then back to writable */+if(mprotect(shstk_ptr,SS_SIZE,PROT_WRITE|PROT_READ)<0){+printf("[FAIL]\tmprotect(PROT_WRITE) failed\n");+return1;+}++/* then wrss to it and succeed */+if(test_shstk_access(shstk_ptr)){+printf("[FAIL]\tShadow stack access to mprotect() writable memory failed\n");+return1;+}++free_shstk(shstk_ptr);++signal(SIGSEGV,SIG_DFL);++printf("[OK]\tmprotect() test\n");++return0;+}++charzero[4096];++staticvoid*uffd_thread(void*arg)+{+structuffdio_copyreq;+intuffd=*(int*)arg;+structuffd_msgmsg;++if(read(uffd,&msg,sizeof(msg))<=0)+return(void*)1;++req.dst=msg.arg.pagefault.address;+req.src=(__u64)zero;+req.len=4096;+req.mode=0;++if(ioctl(uffd,UFFDIO_COPY,&req))+return(void*)1;++return(void*)0;+}++inttest_userfaultfd(void)+{+structuffdio_registeruffdio_register;+structuffdio_apiuffdio_api;+structsigactionsa;+pthread_tthread;+void*res;+intuffd;++sa.sa_sigaction=test_access_fix_handler;+if(sigaction(SIGSEGV,&sa,NULL))+return1;+sa.sa_flags=SA_SIGINFO;++uffd=syscall(__NR_userfaultfd,O_CLOEXEC|O_NONBLOCK);+if(uffd<0){+printf("[SKIP]\tUserfaultfd unavailable.\n");+return0;+}++reset_test_shstk(0);++uffdio_api.api=UFFD_API;+uffdio_api.features=0;+if(ioctl(uffd,UFFDIO_API,&uffdio_api))+gotoerr;++uffdio_register.range.start=(__u64)shstk_ptr;+uffdio_register.range.len=4096;+uffdio_register.mode=UFFDIO_REGISTER_MODE_MISSING;+if(ioctl(uffd,UFFDIO_REGISTER,&uffdio_register))+gotoerr;++if(pthread_create(&thread,NULL,&uffd_thread,&uffd))+gotoerr;++reset_shstk(shstk_ptr);+test_shstk_access(shstk_ptr);++if(pthread_join(thread,&res))+gotoerr;++if(test_shstk_access(shstk_ptr))+gotoerr;++free_shstk(shstk_ptr);++signal(SIGSEGV,SIG_DFL);++if(!res)+printf("[OK]\tUserfaultfd test\n");+return!!res;+err:+free_shstk(shstk_ptr);+close(uffd);+signal(SIGSEGV,SIG_DFL);+return1;+}++/*+*Toocomplicatedtopullitoutofthe32bitheader,butalsogetthe+*64bitoneneededabove.Justdefineacopyhere.+*/+#define __NR_compat_sigaction 67++/*+*Call32bitsignalhandlertoget32bitsignalsABI.Makesure+*topushtheregistersthatwillgetclobbered.+*/+intsigaction32(intsignum,conststructsigaction*restrictact,+structsigaction*restrictoldact)+{+registerlongsyscall_regasm("eax")=__NR_compat_sigaction;+registerlongsignum_regasm("ebx")=signum;+registerlongact_regasm("ecx")=(long)act;+registerlongoldact_regasm("edx")=(long)oldact;+intret=0;++asmvolatile("int $0x80;"+:"=a"(ret),"=m"(oldact)+:"r"(syscall_reg),"r"(signum_reg),"r"(act_reg),+"r"(oldact_reg)+:"r8","r9","r10","r11"+);++returnret;+}++sigjmp_bufjmp_buffer;++voidsegv_gp_handler(intsignum,siginfo_t*si,void*uc)+{+segv_triggered=true;++/*+*Toworkwitholdglibc,thiscan'trelyonsiglongjmpworkingwith+*shadowstackenabled,sodisableshadowstackbeforesiglongjmp().+*/+ARCH_PRCTL(ARCH_SHSTK_DISABLE,ARCH_SHSTK_SHSTK);+siglongjmp(jmp_buffer,-1);+}++/*+*Transitionto32bitmodeandcheckthata#GPtriggersasegfault.+*/+inttest_32bit(void)+{+structsigactionsa;+structsigaction*sa32;++/* Create sigaction in 32 bit address range */+sa32=mmap(0,4096,PROT_READ|PROT_WRITE,+MAP_32BIT|MAP_PRIVATE|MAP_ANONYMOUS,0,0);+sa32->sa_flags=SA_SIGINFO;++sa.sa_sigaction=segv_gp_handler;+if(sigaction(SIGSEGV,&sa,NULL))+return1;+sa.sa_flags=SA_SIGINFO;++segv_triggered=false;++/* Make sure segv_triggered is set before triggering the #GP */+asmvolatile("":::"memory");++/*+*Sethandlertosomewherein32bitaddressspace+*/+sa32->sa_handler=(void*)sa32;+if(sigaction32(SIGUSR1,sa32,NULL))+return1;++if(!sigsetjmp(jmp_buffer,1))+raise(SIGUSR1);++if(segv_triggered)+printf("[OK]\t32 bit test\n");++return!segv_triggered;+}++intmain(intargc,char*argv[])+{+intret=0;++if(ARCH_PRCTL(ARCH_SHSTK_ENABLE,ARCH_SHSTK_SHSTK)){+printf("[SKIP]\tCould not enable Shadow stack\n");+return1;+}++if(ARCH_PRCTL(ARCH_SHSTK_DISABLE,ARCH_SHSTK_SHSTK)){+ret=1;+printf("[FAIL]\tDisabling shadow stack failed\n");+}++if(ARCH_PRCTL(ARCH_SHSTK_ENABLE,ARCH_SHSTK_SHSTK)){+printf("[SKIP]\tCould not re-enable Shadow stack\n");+return1;+}++if(ARCH_PRCTL(ARCH_SHSTK_ENABLE,ARCH_SHSTK_WRSS)){+printf("[SKIP]\tCould not enable WRSS\n");+ret=1;+gotoout;+}++/* Should have succeeded if here, but this is a test, so double check. */+if(!get_ssp()){+printf("[FAIL]\tShadow stack disabled\n");+return1;+}++if(test_shstk_pivot()){+ret=1;+printf("[FAIL]\tShadow stack pivot\n");+gotoout;+}++if(test_shstk_faults()){+ret=1;+printf("[FAIL]\tShadow stack fault test\n");+gotoout;+}++if(test_shstk_violation()){+ret=1;+printf("[FAIL]\tShadow stack violation test\n");+gotoout;+}++if(test_gup()){+ret=1;+printf("[FAIL]\tShadow shadow stack gup\n");+gotoout;+}++if(test_mprotect()){+ret=1;+printf("[FAIL]\tShadow shadow mprotect test\n");+gotoout;+}++if(test_userfaultfd()){+ret=1;+printf("[FAIL]\tUserfaultfd test\n");+gotoout;+}++if(test_32bit()){+ret=1;+printf("[FAIL]\t32 bit test\n");+}++returnret;++out:+/*+*Disableshadowstackbeforethefunctionreturns,ortherewillbea+*shadowstackviolation.+*/+if(ARCH_PRCTL(ARCH_SHSTK_DISABLE,ARCH_SHSTK_SHSTK)){+ret=1;+printf("[FAIL]\tDisabling shadow stack failed\n");+}++returnret;+}+#endif
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:23:17
From: Yu-cheng Yu <redacted>
Some applications (like GDB) would like to tweak shadow stack state via
ptrace. This allows for existing functionality to continue to work for
seized shadow stack applications. Provide an regset interface for
manipulating the shadow stack pointer (SSP).
There is already ptrace functionality for accessing xstate, but this
does not include supervisor xfeatures. So there is not a completely
clear place for where to put the shadow stack state. Adding it to the
user xfeatures regset would complicate that code, as it currently shares
logic with signals which should not have supervisor features.
Don't add a general supervisor xfeature regset like the user one,
because it is better to maintain flexibility for other supervisor
xfeatures to define their own interface. For example, an xfeature may
decide not to expose all of it's state to userspace, as is actually the
case for shadow stack ptrace functionality. A lot of enum values remain
to be used, so just put it in dedicated shadow stack regset.
The only downside to not having a generic supervisor xfeature regset,
is that apps need to be enlightened of any new supervisor xfeature
exposed this way (i.e. they can't try to have generic save/restore
logic). But maybe that is a good thing, because they have to think
through each new xfeature instead of encountering issues when new a new
supervisor xfeature was added.
By adding a shadow stack regset, it also has the effect of including the
shadow stack state in a core dump, which could be useful for debugging.
The shadow stack specific xstate includes the SSP, and the shadow stack
and WRSS enablement status. Enabling shadow stack or wrss in the kernel
involves more than just flipping the bit. The kernel is made aware that
it has to do extra things when cloning or handling signals. That logic
is triggered off of separate feature enablement state kept in the task
struct. So the flipping on HW shadow stack enforcement without notifying
the kernel to change its behavior would severely limit what an application
could do without crashing, and the results would depend on kernel
internal implementation details. There is also no known use for controlling
this state via prtace today. So only expose the SSP, which is something
that userspace already has indirect control over.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Yu-cheng Yu <redacted>
---
v5:
- Check shadow stack enablement status for tracee (rppt)
- Fix typo in comment
v4:
- Make shadow stack only. Reduce to only supporting SSP register, and
remove CET references (peterz)
- Add comment to not use 0x203, because binutils already looks for it in
coredumps. (Christina Schimpe)
v3:
- Drop dependence on thread.shstk.size, and use thread.features bits
- Drop 32 bit support
v2:
- Check alignment on ssp.
- Block IBT bits.
- Handle init states instead of returning error.
- Add verbose commit log justifying the design.
---
arch/x86/include/asm/fpu/regset.h | 7 +--
arch/x86/kernel/fpu/regset.c | 86 +++++++++++++++++++++++++++++++
arch/x86/kernel/ptrace.c | 12 +++++
include/uapi/linux/elf.h | 2 +
4 files changed, 104 insertions(+), 3 deletions(-)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:23:20
From: Mike Rapoport <redacted>
Userspace loaders may lock features before a CRIU restore operation has
the chance to set them to whatever state is required by the process
being restored. Allow a way for CRIU to unlock features. Add it as an
arch_prctl() like the other shadow stack operations, but restrict it being
called by the ptrace arch_pctl() interface.
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Mike Rapoport <redacted>
[Merged into recent API changes, added commit log and docs]
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v4:
- Add to docs that it is ptrace only.
- Remove "CET" references
v3:
- Depend on CONFIG_CHECKPOINT_RESTORE (Kees)
---
Documentation/x86/shstk.rst | 4 ++++
arch/x86/include/uapi/asm/prctl.h | 1 +
arch/x86/kernel/process_64.c | 1 +
arch/x86/kernel/shstk.c | 9 +++++++--
4 files changed, 13 insertions(+), 2 deletions(-)
@@ -73,6 +73,10 @@ arch_prctl(ARCH_SHSTK_LOCK, unsigned long features) are ignored. The mask is ORed with the existing value. So any feature bits set here cannot be enabled or disabled afterwards.+arch_prctl(ARCH_SHSTK_UNLOCK, unsigned long features)+ Unlock features. 'features' is a mask of all features to unlock. All+ bits set are processed, unset bits are ignored. Only works via ptrace.+ The return values are as follows. On success, return 0. On error, errno can be::
@@ -835,6 +835,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)caseARCH_SHSTK_ENABLE:caseARCH_SHSTK_DISABLE:caseARCH_SHSTK_LOCK:+caseARCH_SHSTK_UNLOCK:returnshstk_prctl(task,option,arg2);default:ret=-EINVAL;
@@ -451,9 +451,14 @@ long shstk_prctl(struct task_struct *task, int option, unsigned long features)return0;}-/* Don't allow via ptrace */-if(task!=current)+/* Only allow via ptrace */+if(task!=current){+if(option==ARCH_SHSTK_UNLOCK&&IS_ENABLED(CONFIG_CHECKPOINT_RESTORE)){+task->thread.features_locked&=~features;+return0;+}return-EINVAL;+}/* Do not allow to change locked features */if(features&task->thread.features_locked)
From: Rick Edgecombe <rick.p.edgecombe@intel.com> Date: 2023-02-18 21:23:37
CRIU and GDB need to get the current shadow stack and WRSS enablement
status. This information is already available via /proc/pid/status, but
this is inconvenient for CRIU because it involves parsing the text output
in an area of the code where this is difficult. Provide a status
arch_prctl(), ARCH_SHSTK_STATUS for retrieving the status. Have arg2 be a
userspace address, and make the new arch_prctl simply copy the features
out to userspace.
Tested-by: Pengfei Xu <redacted>
Reviewed-by: Kees Cook <redacted>
Suggested-by: Mike Rapoport <rppt@kernel.org>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v5:
- Fix typo in commit log
v4:
- New patch
---
Documentation/x86/shstk.rst | 6 ++++++
arch/x86/include/asm/shstk.h | 2 +-
arch/x86/include/uapi/asm/prctl.h | 1 +
arch/x86/kernel/process_64.c | 1 +
arch/x86/kernel/shstk.c | 8 +++++++-
5 files changed, 16 insertions(+), 2 deletions(-)
@@ -77,6 +77,11 @@ arch_prctl(ARCH_SHSTK_UNLOCK, unsigned long features) Unlock features. 'features' is a mask of all features to unlock. All bits set are processed, unset bits are ignored. Only works via ptrace.+arch_prctl(ARCH_SHSTK_STATUS, unsigned long addr)+ Copy the currently enabled features to the address passed in addr. The+ features are described using the bits passed into the others in+ 'features'.+ The return values are as follows. On success, return 0. On error, errno can be::
@@ -84,6 +89,7 @@ be:: -ENOTSUPP if the feature is not supported by the hardware or kernel. -EINVAL arguments (non existing feature, etc)+ -EFAULT if could not copy information back to userspace The feature's bits supported are::
@@ -836,6 +836,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)caseARCH_SHSTK_DISABLE:caseARCH_SHSTK_LOCK:caseARCH_SHSTK_UNLOCK:+caseARCH_SHSTK_STATUS:returnshstk_prctl(task,option,arg2);default:ret=-EINVAL;
On Sat, Feb 18, 2023 at 01:14:03PM -0800, Rick Edgecombe wrote:
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One of these changes is to allow for pte_mkwrite() to create different
types of writable memory (the existing conventionally writable type and
also the new shadow stack type). Future patches will convert pte_mkwrite()
to take a VMA in order to facilitate this, however there are places in the
kernel where pte_mkwrite() is called outside of the context of a VMA.
These are for kernel memory. So create a new variant called
pte_mkwrite_kernel() and switch the kernel users over to it. Have
pte_mkwrite() and pte_mkwrite_kernel() be the same for now. Future patches
will introduce changes to make pte_mkwrite() take a VMA.
Only do this for architectures that need it because they call pte_mkwrite()
in arch code without an associated VMA. Since it will only currently be
used in arch code, so do not include it in arch_pgtable_helpers.rst.
Cc: linux-doc@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-s390@vger.kernel.org
Cc: xen-devel@lists.xenproject.org
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Tested-by: Pengfei Xu <redacted>
Suggested-by: David Hildenbrand <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
I think it's a little weird that it's the only PTE helper taking a vma,
but it does seem like the right approach.
Reviewed-by: Kees Cook <redacted>
--
Kees Cook
On Sat, Feb 18, 2023 at 01:14:04PM -0800, Rick Edgecombe wrote:
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One of these changes is to allow for pmd_mkwrite() to create different
types of writable memory (the existing conventionally writable type and
also the new shadow stack type). Future patches will convert pmd_mkwrite()
to take a VMA in order to facilitate this, however there are places in the
kernel where pmd_mkwrite() is called outside of the context of a VMA.
These are for kernel memory. So create a new variant called
pmd_mkwrite_kernel() and switch the kernel users over to it. Have
pmd_mkwrite() and pmd_mkwrite_kernel() be the same for now. Future patches
will introduce changes to make pmd_mkwrite() take a VMA.
Only do this for architectures that need it because they call pmd_mkwrite()
in arch code without an associated VMA. Since it will only currently be
used in arch code, so do not include it in arch_pgtable_helpers.rst.
Cc: linux-kernel@vger.kernel.org
Cc: linux-s390@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Tested-by: Pengfei Xu <redacted>
Suggested-by: David Hildenbrand <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
On Sat, Feb 18, 2023 at 01:14:05PM -0800, Rick Edgecombe wrote:
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One of these unusual properties is that shadow stack memory is writable,
but only in limited ways. These limits are applied via a specific PTE
bit combination. Nevertheless, the memory is writable, and core mm code
will need to apply the writable permissions in the typical paths that
call pte_mkwrite().
In addition to VM_WRITE, the shadow stack VMA's will have a flag denoting
that they are special shadow stack flavor of writable memory. So make
pte_mkwrite() take a VMA, so that the x86 implementation of it can know to
create regular writable memory or shadow stack memory.
Apply the same changes for pmd_mkwrite() and huge_pte_mkwrite().
No functional change.
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-alpha@vger.kernel.org
Cc: linux-snps-arc@lists.infradead.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-csky@vger.kernel.org
Cc: linux-hexagon@vger.kernel.org
Cc: linux-ia64@vger.kernel.org
Cc: loongarch@lists.linux.dev
Cc: linux-m68k@lists.linux-m68k.org
Cc: Michal Simek <monstr@monstr.eu>
Cc: Dinh Nguyen <dinguyen@kernel.org>
Cc: linux-mips@vger.kernel.org
Cc: openrisc@lists.librecores.org
Cc: linux-parisc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-riscv@lists.infradead.org
Cc: linux-s390@vger.kernel.org
Cc: linux-sh@vger.kernel.org
Cc: sparclinux@vger.kernel.org
Cc: linux-um@lists.infradead.org
Cc: xen-devel@lists.xenproject.org
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Tested-by: Pengfei Xu <redacted>
Suggested-by: David Hildenbrand <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
I'm not an arch maintainer, but it looks like a correct tree-wide
refactor.
Reviewed-by: Kees Cook <redacted>
--
Kees Cook
On Sat, Feb 18, 2023 at 01:14:12PM -0800, Rick Edgecombe wrote:
If a VMA has the VM_SHADOW_STACK flag, it is shadow stack memory. So
when it is made writable with pte_mkwrite(), it should create shadow
stack memory, not conventionally writable memory. Now that pte_mkwrite()
takes a VMA, and places where shadow stack memory might be created pass
one, pte_mkwrite() can know when it should do this.
So make pte_mkwrite() create shadow stack memory when the VMA has the
VM_SHADOW_STACK flag. Do the same thing for pmd_mkwrite().
This requires referencing VM_SHADOW_STACK in these functions, which are
currently defined in pgtable.h, however mm.h (where VM_SHADOW_STACK is
located) can't be pulled in without causing problems for files that
reference pgtable.h. So also move pte/pmd_mkwrite() into pgtable.c, where
they can safely reference VM_SHADOW_STACK.
Tested-by: Pengfei Xu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Is there any realistic performance impact from making these not inline
now?
Reviewed-by: Kees Cook <redacted>
--
Kees Cook
On Sat, Feb 18, 2023 at 01:14:17PM -0800, Rick Edgecombe wrote:
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which require some core mm changes to function
properly.
One of the properties is that the shadow stack pointer (SSP), which is a
CPU register that points to the shadow stack like the stack pointer points
to the stack, can't be pointing outside of the 32 bit address space when
the CPU is executing in 32 bit mode. It is desirable to prevent executing
in 32 bit mode when shadow stack is enabled because the kernel can't easily
support 32 bit signals.
On x86 it is possible to transition to 32 bit mode without any special
interaction with the kernel, by doing a "far call" to a 32 bit segment.
So the shadow stack implementation can use this address space behavior
as a feature, by enforcing that shadow stack memory is always crated
outside of the 32 bit address space. This way userspace will trigger a
general protection fault which will in turn trigger a segfault if it
tries to transition to 32 bit mode with shadow stack enabled.
This provides a clean error generating border for the user if they try
attempt to do 32 bit mode shadow stack, rather than leave the kernel in a
half working state for userspace to be surprised by.
So to allow future shadow stack enabling patches to map shadow stacks
out of the 32 bit address space, introduce MAP_ABOVE4G. The behavior
is pretty much like MAP_32BIT, except that it has the opposite address
range. The are a few differences though.
If both MAP_32BIT and MAP_ABOVE4G are provided, the kernel will use the
MAP_ABOVE4G behavior. Like MAP_32BIT, MAP_ABOVE4G is ignored in a 32 bit
syscall.
Should the interface refuse to accept both set instead?
Reviewed-by: Kees Cook <redacted>
--
Kees Cook
On Sat, Feb 18, 2023 at 01:14:19PM -0800, Rick Edgecombe wrote:
quoted hunk
When user shadow stack is use, Write=0,Dirty=1 is treated by the CPU as
shadow stack memory. So for shadow stack memory this bit combination is
valid, but when Dirty=1,Write=1 (conventionally writable) memory is being
write protected, the kernel has been taught to transition the Dirty=1
bit to SavedDirty=1, to avoid inadvertently creating shadow stack
memory. It does this inside pte_wrprotect() because it knows the PTE is
not intended to be a writable shadow stack entry, it is supposed to be
write protected.
However, when a PTE is created by a raw prot using mk_pte(), mk_pte()
can't know whether to adjust Dirty=1 to SavedDirty=1. It can't
distinguish between the caller intending to create a shadow stack PTE or
needing the SavedDirty shift.
The kernel has been updated to not do this, and so Write=0,Dirty=1
memory should only be created by the pte_mkfoo() helpers. Add a warning
to make sure no new mk_pte() start doing this.
Tested-by: Pengfei Xu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- New patch (Note, this has already been a useful warning, it caught the
newly added set_memory_rox() doing this)
---
arch/x86/include/asm/pgtable.h | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
On Sat, Feb 18, 2023 at 01:14:29PM -0800, Rick Edgecombe wrote:
Add a simple selftest for exercising some shadow stack behavior:
- map_shadow_stack syscall and pivot
- Faulting in shadow stack memory
- Handling shadow stack violations
- GUP of shadow stack memory
- mprotect() of shadow stack memory
- Userfaultfd on shadow stack memory
Since this test exercises a recently added syscall manually, it needs
to find the automatically created __NR_foo defines. Per the selftest
documentation, KHDR_INCLUDES can be used to help the selftest Makefile's
find the headers from the kernel source. This way the new selftest can
be built inside the kernel source tree without installing the headers
to the system. So also add KHDR_INCLUDES as described in the selftest
docs, to facilitate this.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Co-developed-by: Yu-cheng Yu <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
I'll get some test hardware and run this myself too, but overall,
ignoring the lack of kselftest_harness.h, it looks good:
Reviewed-by: Kees Cook <redacted>
--
Kees Cook
On Sat, Feb 18, 2023 at 01:14:30PM -0800, Rick Edgecombe wrote:
If an xfeature is saved in a buffer, the xfeature's bit will be set in
xsave->header.xfeatures. The CPU may opt to not save the xfeature if it
is in it's init state. In this case the xfeature buffer address cannot
be retrieved with get_xsave_addr().
Future patches will need to handle the case of writing to an xfeature
that may not be saved. So provide helpers to init an xfeature in an
xsave buffer.
This could of course be done directly by reaching into the xsave buffer,
however this would not be robust against future changes to optimize the
xsave buffer by compacting it. In that case the xsave buffer would need
to be re-arranged as well. So the logic properly belongs encapsulated
in a helper where the logic can be unified.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2023-02-20 01:01:06
Rick Edgecombe [off-list ref] writes:
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
...
---
Hi Non-x86 Arch’s,
x86 has a feature that allows for the creation of a special type of
writable memory (shadow stack) that is only writable in limited specific
ways. Previously, changes were proposed to core MM code to teach it to
decide when to create normally writable memory or the special shadow stack
writable memory, but David Hildenbrand suggested[0] to change
pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be
moved into x86 code.
Since pXX_mkwrite() is defined in every arch, it requires some tree-wide
changes. So that is why you are seeing some patches out of a big x86
series pop up in your arch mailing list. There is no functional change.
After this refactor, the shadow stack series goes on to use the arch
helpers to push shadow stack memory details inside arch/x86.
Looks like you discovered the joys of ppc's at-least 5 different MMU
implementations, sorry :)
Acked-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc)
cheers
On Sat, Feb 18, 2023 at 01:13:52PM -0800, Rick Edgecombe wrote:
This series implements Shadow Stacks for userspace using x86's Control-flow
Enforcement Technology (CET). CET consists of two related security features:
shadow stacks and indirect branch tracking. This series implements just the
shadow stack part of this feature, and just for userspace.
Okay, I've done some bare metal testing, and it all looks happy. The
selftest passes, and I can can see the stack address mismatch get
detected if I explicitly rewrite the saved function pointer on the stack:
[INFO] Want normal flow
[INFO] Found 0x401890 @ 0x7fff47cf2ef8
[INFO] Normal execution flow
[INFO] Want to redirect
[INFO] Found 0x401890 @ 0x7fff47cf2ef8
[INFO] Hijacked execution flow
[INFO] Enabling shadow stack
[INFO] Want to redirect
[INFO] Found 0x401890 @ 0x7fff47cf2ef8
Segmentation fault (core dumped)
Tested-by: Kees Cook <redacted>
--
Kees Cook
From: Mike Rapoport <rppt@kernel.org> Date: 2023-02-20 06:50:58
On Sat, Feb 18, 2023 at 01:13:52PM -0800, Rick Edgecombe wrote:
Hi,
This series implements Shadow Stacks for userspace using x86's Control-flow
Enforcement Technology (CET). CET consists of two related security features:
shadow stacks and indirect branch tracking. This series implements just the
shadow stack part of this feature, and just for userspace.
For the series
Acked-by: Mike Rapoport (IBM) <rppt@kernel.org>
--
Sincerely yours,
Mike.
From: David Hildenbrand <hidden> Date: 2023-02-20 11:19:01
On 19.02.23 21:38, Kees Cook wrote:
On Sat, Feb 18, 2023 at 01:14:03PM -0800, Rick Edgecombe wrote:
quoted
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One of these changes is to allow for pte_mkwrite() to create different
types of writable memory (the existing conventionally writable type and
also the new shadow stack type). Future patches will convert pte_mkwrite()
to take a VMA in order to facilitate this, however there are places in the
kernel where pte_mkwrite() is called outside of the context of a VMA.
These are for kernel memory. So create a new variant called
pte_mkwrite_kernel() and switch the kernel users over to it. Have
pte_mkwrite() and pte_mkwrite_kernel() be the same for now. Future patches
will introduce changes to make pte_mkwrite() take a VMA.
Only do this for architectures that need it because they call pte_mkwrite()
in arch code without an associated VMA. Since it will only currently be
used in arch code, so do not include it in arch_pgtable_helpers.rst.
Cc: linux-doc@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-s390@vger.kernel.org
Cc: xen-devel@lists.xenproject.org
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Tested-by: Pengfei Xu <redacted>
Suggested-by: David Hildenbrand <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
I think it's a little weird that it's the only PTE helper taking a vma,
but it does seem like the right approach.
Right. We could pass the vm flags instead, but not sure if that really
improves the situation. So unless someone has a better idea, this LGTM.
--
Thanks,
David / dhildenb
From: David Hildenbrand <hidden> Date: 2023-02-20 11:20:46
On 18.02.23 22:14, Rick Edgecombe wrote:
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One of these changes is to allow for pte_mkwrite() to create different
types of writable memory (the existing conventionally writable type and
also the new shadow stack type). Future patches will convert pte_mkwrite()
to take a VMA in order to facilitate this, however there are places in the
kernel where pte_mkwrite() is called outside of the context of a VMA.
These are for kernel memory. So create a new variant called
pte_mkwrite_kernel() and switch the kernel users over to it. Have
pte_mkwrite() and pte_mkwrite_kernel() be the same for now. Future patches
will introduce changes to make pte_mkwrite() take a VMA.
Only do this for architectures that need it because they call pte_mkwrite()
in arch code without an associated VMA. Since it will only currently be
used in arch code, so do not include it in arch_pgtable_helpers.rst.
Cc: linux-doc@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-s390@vger.kernel.org
Cc: xen-devel@lists.xenproject.org
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Tested-by: Pengfei Xu <redacted>
Suggested-by: David Hildenbrand <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Acked-by: David Hildenbrand <redacted>
Do we also have to care about pmd_mkwrite() ?
--
Thanks,
David / dhildenb
From: David Hildenbrand <hidden> Date: 2023-02-20 11:22:06
On 18.02.23 22:14, Rick Edgecombe wrote:
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One of these changes is to allow for pmd_mkwrite() to create different
types of writable memory (the existing conventionally writable type and
also the new shadow stack type). Future patches will convert pmd_mkwrite()
to take a VMA in order to facilitate this, however there are places in the
kernel where pmd_mkwrite() is called outside of the context of a VMA.
These are for kernel memory. So create a new variant called
pmd_mkwrite_kernel() and switch the kernel users over to it. Have
pmd_mkwrite() and pmd_mkwrite_kernel() be the same for now. Future patches
will introduce changes to make pmd_mkwrite() take a VMA.
Only do this for architectures that need it because they call pmd_mkwrite()
in arch code without an associated VMA. Since it will only currently be
used in arch code, so do not include it in arch_pgtable_helpers.rst.
Cc: linux-kernel@vger.kernel.org
Cc: linux-s390@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Tested-by: Pengfei Xu <redacted>
Suggested-by: David Hildenbrand <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Heh, that answers my question to patch #11
Acked-by: David Hildenbrand <redacted>
--
Thanks,
David / dhildenb
From: David Hildenbrand <hidden> Date: 2023-02-20 11:24:33
On 18.02.23 22:14, Rick Edgecombe wrote:
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
One of these unusual properties is that shadow stack memory is writable,
but only in limited ways. These limits are applied via a specific PTE
bit combination. Nevertheless, the memory is writable, and core mm code
will need to apply the writable permissions in the typical paths that
call pte_mkwrite().
In addition to VM_WRITE, the shadow stack VMA's will have a flag denoting
that they are special shadow stack flavor of writable memory. So make
pte_mkwrite() take a VMA, so that the x86 implementation of it can know to
create regular writable memory or shadow stack memory.
Apply the same changes for pmd_mkwrite() and huge_pte_mkwrite().
No functional change.
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-alpha@vger.kernel.org
Cc: linux-snps-arc@lists.infradead.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-csky@vger.kernel.org
Cc: linux-hexagon@vger.kernel.org
Cc: linux-ia64@vger.kernel.org
Cc: loongarch@lists.linux.dev
Cc: linux-m68k@lists.linux-m68k.org
Cc: Michal Simek <monstr@monstr.eu>
Cc: Dinh Nguyen <dinguyen@kernel.org>
Cc: linux-mips@vger.kernel.org
Cc: openrisc@lists.librecores.org
Cc: linux-parisc@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-riscv@lists.infradead.org
Cc: linux-s390@vger.kernel.org
Cc: linux-sh@vger.kernel.org
Cc: sparclinux@vger.kernel.org
Cc: linux-um@lists.infradead.org
Cc: xen-devel@lists.xenproject.org
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Tested-by: Pengfei Xu <redacted>
Suggested-by: David Hildenbrand <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
Hi Non-x86 Arch’s,
x86 has a feature that allows for the creation of a special type of
writable memory (shadow stack) that is only writable in limited specific
ways. Previously, changes were proposed to core MM code to teach it to
decide when to create normally writable memory or the special shadow stack
writable memory, but David Hildenbrand suggested[0] to change
pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be
moved into x86 code.
Since pXX_mkwrite() is defined in every arch, it requires some tree-wide
changes. So that is why you are seeing some patches out of a big x86
series pop up in your arch mailing list. There is no functional change.
After this refactor, the shadow stack series goes on to use the arch
helpers to push shadow stack memory details inside arch/x86.
Testing was just 0-day build testing.
Hopefully that is enough context. Thanks!
[0] https://lore.kernel.org/lkml/0e29a2d0-08d8-bcd6-ff26-4bea0e4037b0@redhat.com/#t
v6:
- New patch
---
Documentation/mm/arch_pgtable_helpers.rst | 9 ++++++---
arch/alpha/include/asm/pgtable.h | 6 +++++-
arch/arc/include/asm/hugepage.h | 2 +-
arch/arc/include/asm/pgtable-bits-arcv2.h | 7 ++++++-
arch/arm/include/asm/pgtable-3level.h | 7 ++++++-
arch/arm/include/asm/pgtable.h | 2 +-
arch/arm64/include/asm/pgtable.h | 4 ++--
arch/csky/include/asm/pgtable.h | 2 +-
arch/hexagon/include/asm/pgtable.h | 2 +-
arch/ia64/include/asm/pgtable.h | 2 +-
arch/loongarch/include/asm/pgtable.h | 4 ++--
arch/m68k/include/asm/mcf_pgtable.h | 2 +-
arch/m68k/include/asm/motorola_pgtable.h | 6 +++++-
arch/m68k/include/asm/sun3_pgtable.h | 6 +++++-
arch/microblaze/include/asm/pgtable.h | 2 +-
arch/mips/include/asm/pgtable.h | 6 +++---
arch/nios2/include/asm/pgtable.h | 2 +-
arch/openrisc/include/asm/pgtable.h | 2 +-
arch/parisc/include/asm/pgtable.h | 6 +++++-
arch/powerpc/include/asm/book3s/32/pgtable.h | 2 +-
arch/powerpc/include/asm/book3s/64/pgtable.h | 4 ++--
arch/powerpc/include/asm/nohash/32/pgtable.h | 2 +-
arch/powerpc/include/asm/nohash/32/pte-8xx.h | 2 +-
arch/powerpc/include/asm/nohash/64/pgtable.h | 2 +-
arch/riscv/include/asm/pgtable.h | 6 +++---
arch/s390/include/asm/hugetlb.h | 4 ++--
arch/s390/include/asm/pgtable.h | 4 ++--
arch/sh/include/asm/pgtable_32.h | 10 ++++++++--
arch/sparc/include/asm/pgtable_32.h | 2 +-
arch/sparc/include/asm/pgtable_64.h | 6 +++---
arch/um/include/asm/pgtable.h | 2 +-
arch/x86/include/asm/pgtable.h | 6 ++++--
arch/xtensa/include/asm/pgtable.h | 2 +-
include/asm-generic/hugetlb.h | 4 ++--
include/linux/mm.h | 2 +-
mm/debug_vm_pgtable.c | 16 ++++++++--------
mm/huge_memory.c | 6 +++---
mm/hugetlb.c | 4 ++--
mm/memory.c | 4 ++--
mm/migrate_device.c | 2 +-
mm/mprotect.c | 2 +-
mm/userfaultfd.c | 2 +-
42 files changed, 106 insertions(+), 69 deletions(-)
That looks painful but IMHO worth it :)
Acked-by: David Hildenbrand <redacted>
--
Thanks,
David / dhildenb
From: David Hildenbrand <hidden> Date: 2023-02-20 11:32:50
On 18.02.23 22:14, Rick Edgecombe wrote:
Some OSes have a greater dependence on software available bits in PTEs than
Linux. That left the hardware architects looking for a way to represent a
new memory type (shadow stack) within the existing bits. They chose to
repurpose a lightly-used state: Write=0,Dirty=1. So in order to support
shadow stack memory, Linux should avoid creating memory with this PTE bit
combination unless it intends for it to be shadow stack.
The reason it's lightly used is that Dirty=1 is normally set by HW
_before_ a write. A write with a Write=0 PTE would typically only generate
a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and*
generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which
supports shadow stacks will no longer exhibit this oddity.
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this,
in places where Linux normally creates Write=0,Dirty=1, it can use the
software-defined _PAGE_SAVED_DIRTY in place of the hardware _PAGE_DIRTY.
In other words, whenever Linux needs to create Write=0,Dirty=1, it instead
creates Write=0,SavedDirty=1 except for shadow stack, which is
Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit
combinations would be set as follows for various types of memory:
I would simplify (see below) and not repeat what the patch contains as
comments already that detailed.
quoted hunk
(Write=0,SavedDirty=1,Dirty=0):
- A modified, copy-on-write (COW) page. Previously when a typical
anonymous writable mapping was made COW via fork(), the kernel would
mark it Write=0,Dirty=1. Now it will instead use the SavedDirty bit.
This happens in copy_present_pte().
- A R/O page that has been COW'ed. The user page is in a R/O VMA,
and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault
handler creates a copy of the page and sets the new copy's PTE as
Write=0 and SavedDirty=1.
- A shared shadow stack PTE. When a shadow stack page is being shared
among processes (this happens at fork()), its PTE is made Dirty=0, so
the next shadow stack access causes a fault, and the page is
duplicated and Dirty=1 is set again. This is the COW equivalent for
shadow stack pages, even though it's copy-on-access rather than
copy-on-write.
(Write=0,SavedDirty=0,Dirty=1):
- A shadow stack PTE.
- A Cow PTE created when a processor without shadow stack support set
Dirty=1.
There are six bits left available to software in the 64-bit PTE after
consuming a bit for _PAGE_SAVED_DIRTY. No space is consumed in 32-bit
kernels because shadow stacks are not enabled there.
Implement only the infrastructure for _PAGE_SAVED_DIRTY. Changes to start
creating _PAGE_SAVED_DIRTY PTEs will follow once other pieces are in place.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Co-developed-by: Yu-cheng Yu <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Rename _PAGE_COW to _PAGE_SAVED_DIRTY (David Hildenbrand)
- Add _PAGE_SAVED_DIRTY to _PAGE_CHG_MASK
v5:
- Fix log, comments and whitespace (Boris)
- Remove capitalization on shadow stack (Boris)
v4:
- Teach pte_flags_need_flush() about _PAGE_COW bit
- Break apart patch for better bisectability
v3:
- Add comment around _PAGE_TABLE in response to comment
from (Andrew Cooper)
- Check for PSE in pmd_shstk (Andrew Cooper)
- Get to the point quicker in commit log (Andrew Cooper)
- Clarify and reorder commit log for why the PTE bit examples have
multiple entries. Apply same changes for comment. (peterz)
- Fix comment that implied dirty bit for COW was a specific x86 thing
(peterz)
- Fix swapping of Write/Dirty (PeterZ)
---
arch/x86/include/asm/pgtable.h | 79 ++++++++++++++++++++++++++++
arch/x86/include/asm/pgtable_types.h | 65 ++++++++++++++++++++---
arch/x86/include/asm/tlbflush.h | 3 +-
3 files changed, 138 insertions(+), 9 deletions(-)
Nope, not "copy-on-write" :) It's more like "dirty bit when the hw-dirty
bit cannot be used". Maybe simply drop the comment.
quoted hunk
+#else
+#define _PAGE_BIT_SAVED_DIRTY 0
+#endif
+
/* If _PAGE_BIT_PRESENT is clear, we use these: */
/* - if the user mapped it with PROT_NONE; pte_present gives true */
#define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL
@@ -117,6 +127,40 @@ #define _PAGE_SOFTW4 (_AT(pteval_t, 0)) #endif+/*+ * The hardware requires shadow stack to be read-only and Dirty.+ * _PAGE_SAVED_DIRTY is a software-only bit used to separate copy-on-write+ * PTEs from shadow stack PTEs:
I'd suggest phrasing this differently. COW is just one scenario where
this can happen. Also, I don't think that the description of
"separation" is correct.
Something like the following maybe?
"
However, there are valid cases where the kernel might create read-only
PTEs that are dirty (e.g., fork(), mprotect(), uffd-wp(), soft-dirty
tracking). In this case, the _PAGE_SAVED_DIRTY bit is used instead of
the HW-dirty bit, to avoid creating a wrong "shadow stack" PTEs. Such
PTEs have (Write=0,SavedDirty=1,Dirty=0) set.
Note that on processors without shadow stack support, the
_PAGE_SAVED_DIRTY remains unused.
"
The I would simply drop below (which is also too COW-specific I think).
+ *
+ * (Write=0,SavedDirty=1,Dirty=0):
+ * - A modified, copy-on-write (COW) page. Previously when a typical
+ * anonymous writable mapping was made COW via fork(), the kernel would
+ * mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This
+ * happens in copy_present_pte().
+ * - A R/O page that has been COW'ed. The user page is in a R/O VMA,
+ * and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault
+ * handler creates a copy of the page and sets the new copy's PTE as
+ * Write=0 and SavedDirty=1.
+ * - A shared shadow stack PTE. When a shadow stack page is being shared
+ * among processes (this happens at fork()), its PTE is made Dirty=0, so
+ * the next shadow stack access causes a fault, and the page is
+ * duplicated and Dirty=1 is set again. This is the COW equivalent for
+ * shadow stack pages, even though it's copy-on-access rather than
+ * copy-on-write.
+ *
+ * (Write=0,SavedDirty=0,Dirty=1):
+ * - A shadow stack PTE.
+ * - A Cow PTE created when a processor without shadow stack support set
+ * Dirty=1.
+ */
From: David Hildenbrand <hidden> Date: 2023-02-20 12:57:02
On 18.02.23 22:14, Rick Edgecombe wrote:
From: Yu-cheng Yu <redacted>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
A shadow stack PTE must be read-only and have _PAGE_DIRTY set. However,
read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These
two cases are handled differently for page faults. Introduce
VM_SHADOW_STACK to track shadow stack VMAs.
I suggest simplifying and abstracting that description.
"New hardware extensions implement support for shadow stack memory, such
as x86 Control-flow Enforcement Technology (CET). Let's add a new VM
flag to identify these areas, for example, to be used to properly
indicate shadow stack PTEs to the hardware."
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Yu-cheng Yu <redacted>
Reviewed-by: Kirill A. Shutemov <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v6:
- Add comment about VM_SHADOW_STACK not being allowed with VM_SHARED
(David Hildenbrand)
Might want to add some more meat to the patch description why that is
the case.
quoted hunk
v3:
- Drop arch specific change in arch_vma_name(). The memory can show as
anonymous (Kirill)
- Change CONFIG_ARCH_HAS_SHADOW_STACK to CONFIG_X86_USER_SHADOW_STACK
in show_smap_vma_flags() (Boris)
---
Documentation/filesystems/proc.rst | 1 +
fs/proc/task_mmu.c | 3 +++
include/linux/mm.h | 8 ++++++++
3 files changed, 12 insertions(+)
@@ -564,6 +564,7 @@ encoded manner. The codes are the following: mt arm64 MTE allocation tags are enabled um userfaultfd missing tracking uw userfaultfd wr-protect tracking+ ss shadow stack page == ======================================= Note that there is no guarantee that every flag and associated mnemonic will
From: David Hildenbrand <hidden> Date: 2023-02-20 12:58:05
On 18.02.23 22:14, Rick Edgecombe wrote:
quoted hunk
From: Yu-cheng Yu <redacted>
The CPU performs "shadow stack accesses" when it expects to encounter
shadow stack mappings. These accesses can be implicit (via CALL/RET
instructions) or explicit (instructions like WRSS).
Shadow stack accesses to shadow-stack mappings can result in faults in
normal, valid operation just like regular accesses to regular mappings.
Shadow stacks need some of the same features like delayed allocation, swap
and copy-on-write. The kernel needs to use faults to implement those
features.
The architecture has concepts of both shadow stack reads and shadow stack
writes. Any shadow stack access to non-shadow stack memory will generate
a fault with the shadow stack error code bit set.
This means that, unlike normal write protection, the fault handler needs
to create a type of memory that can be written to (with instructions that
generate shadow stack writes), even to fulfill a read access. So in the
case of COW memory, the COW needs to take place even with a shadow stack
read. Otherwise the page will be left (shadow stack) writable in
userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE
for shadow stack accesses, even if the access was a shadow stack read.
For the purpose of making this clearer, consider the following example.
If a process has a shadow stack, and forks, the shadow stack PTEs will
become read-only due to COW. If the CPU in one process performs a shadow
stack read access to the shadow stack, for example executing a RET and
causing the CPU to read the shadow stack copy of the return address, then
in order for the fault to be resolved the PTE will need to be set with
shadow stack permissions. But then the memory would be changeable from
userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger
COW, otherwise the shared page would be changeable from both processes.
Shadow stack accesses can also result in errors, such as when a shadow
stack overflows, or if a shadow stack access occurs to a non-shadow-stack
mapping. Also, generate the errors for invalid shadow stack accesses.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Update comment due to rename of Cow bit to SavedDirty
v5:
- Add description of COW example (Boris)
- Replace "permissioned" (Boris)
- Remove capitalization of shadow stack (Boris)
v4:
- Further improve comment talking about FAULT_FLAG_WRITE (Peterz)
v3:
- Improve comment talking about using FAULT_FLAG_WRITE (Peterz)
---
arch/x86/include/asm/trap_pf.h | 2 ++
arch/x86/mm/fault.c | 38 ++++++++++++++++++++++++++++++++++
2 files changed, 40 insertions(+)
@@ -1138,8 +1138,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma)(error_code&X86_PF_INSTR),foreign))return1;+/*+*Shadowstackaccesses(PF_SHSTK=1)areonlypermittedto+*shadowstackVMAs.Allotheraccessesresultinanerror.+*/+if(error_code&X86_PF_SHSTK){+if(unlikely(!(vma->vm_flags&VM_SHADOW_STACK)))+return1;+if(unlikely(!(vma->vm_flags&VM_WRITE)))+return1;+return0;+}+if(error_code&X86_PF_WRITE){/* write, present and write, not present: */+if(unlikely(vma->vm_flags&VM_SHADOW_STACK))+return1;if(unlikely(!(vma->vm_flags&VM_WRITE)))return1;return0;
@@ -1331,6 +1345,30 @@ void do_user_addr_fault(struct pt_regs *regs,perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS,1,regs,address);+/*+*WhenapagebecomesCOWitchangesfromashadowstackpermission+*page(Write=0,Dirty=1)to(Write=0,Dirty=0,SavedDirty=1),whichissimply+*read-onlytotheCPU.Whenshadowstackisenabled,aRETwould+*normallypoptheshadowstackbyreadingitwitha"shadow stack+*read" access. However, in the COW case the shadow stack memory does+*nothaveshadowstackpermissions,itisread-only.Soitwill+*generateafault.+*+*Forconventionallywritablepages,areadcanbeservicedwitha+*readonlyPTE,andCOWwouldnothavetohappen.Butforshadow+*stack,thereisn'ttheconceptofread-onlyshadowstackmemory.+*Ifitisshadowstackpermission,itcanbemodifiedviaCALLand+*RETinstructions.SoCOWneedstohappenbeforeanymemorycanbe+*mappedwithshadowstackpermissions.+*+*Shadowstackaccesses(readorwrite)needtobeservicedwith+*shadowstackpermissionmemory,sointhecaseofashadowstack+*readaccess,treatitasaWRITEfaultsobothCOWwillhappenand+*thewritefaultpathwillticklemaybe_mkwrite()andmapthememory+*shadowstack.+*/
Again, I suggest dropping all details about COW from this comment and
from the patch description. It's just one such case that can happen.
--
Thanks,
David / dhildenb
From: David Hildenbrand <hidden> Date: 2023-02-20 12:59:38
On 18.02.23 22:14, Rick Edgecombe wrote:
quoted hunk
From: Yu-cheng Yu <redacted>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
Account shadow stack pages to stack memory.
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v3:
- Remove unneeded VM_SHADOW_STACK check in accountable_mapping()
(Kirill)
v2:
- Remove is_shadow_stack_mapping() and just change it to directly bitwise
and VM_SHADOW_STACK.
Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.
Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping().
---
mm/mmap.c | 2 ++
1 file changed, 2 insertions(+)
From: John Allen <john.allen@amd.com> Date: 2023-02-20 20:24:38
On Sat, Feb 18, 2023 at 01:13:52PM -0800, Rick Edgecombe wrote:
I left tested-by tags in place per discussion with testers. Testers, please
retest.
v6 is still working well on my AMD system (Dell PowerEdge
R6515 w/ EPYC 7713).
The selftests run cleanly:
[INFO] new_ssp = 7f53069ffff8, *new_ssp = 7f5306a00001
[INFO] changing ssp from 7f53071ffff0 to 7f53069ffff8
[INFO] ssp is now 7f5306a00000
[OK] Shadow stack pivot
[OK] Shadow stack faults
[INFO] Corrupting shadow stack
[INFO] Generated shadow stack violation successfully
[OK] Shadow stack violation test
[INFO] Gup read -> shstk access success
[INFO] Gup write -> shstk access success
[INFO] Violation from normal write
[INFO] Gup read -> write access success
[INFO] Violation from normal write
[INFO] Gup write -> write access success
[INFO] Cow gup write -> write access success
[OK] Shadow gup test
[INFO] Violation from shstk access
[OK] mprotect() test
[OK] Userfaultfd test
[OK] 32 bit test
And I can see the control protection messages in dmesg when
running the shstk violation test from here:
https://gitlab.com/cet-software/cet-smoke-test
ld-linux-x86-64[51598] control protection ip:401139 sp:7ffd68b1b7c8 ssp:7fb433578fd8 error:1(near ret) in shstk1[401000+1000]
Tested-by: John Allen <john.allen@amd.com>
From: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com> Date: 2023-02-20 21:24:02
On Mon, 2023-02-20 at 08:50 +0200, Mike Rapoport wrote:
On Sat, Feb 18, 2023 at 01:13:52PM -0800, Rick Edgecombe wrote:
quoted
Hi,
This series implements Shadow Stacks for userspace using x86's
Control-flow
Enforcement Technology (CET). CET consists of two related security
features:
shadow stacks and indirect branch tracking. This series implements
just the
shadow stack part of this feature, and just for userspace.
For the series
Acked-by: Mike Rapoport (IBM) <rppt@kernel.org>
Thanks Mike! Sorry forgot to add it since last time.
From: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com> Date: 2023-02-20 21:38:45
On Mon, 2023-02-20 at 12:32 +0100, David Hildenbrand wrote:
On 18.02.23 22:14, Rick Edgecombe wrote:
quoted
Some OSes have a greater dependence on software available bits in
PTEs than
Linux. That left the hardware architects looking for a way to
represent a
new memory type (shadow stack) within the existing bits. They chose
to
repurpose a lightly-used state: Write=0,Dirty=1. So in order to
support
shadow stack memory, Linux should avoid creating memory with this
PTE bit
combination unless it intends for it to be shadow stack.
The reason it's lightly used is that Dirty=1 is normally set by HW
_before_ a write. A write with a Write=0 PTE would typically only
generate
a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1
*and*
generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware
which
supports shadow stacks will no longer exhibit this oddity.
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve
this,
in places where Linux normally creates Write=0,Dirty=1, it can use
the
software-defined _PAGE_SAVED_DIRTY in place of the hardware
_PAGE_DIRTY.
In other words, whenever Linux needs to create Write=0,Dirty=1, it
instead
creates Write=0,SavedDirty=1 except for shadow stack, which is
Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit
combinations would be set as follows for various types of memory:
I would simplify (see below) and not repeat what the patch contains
as
comments already that detailed.
This verbiage has had quite a bit of x86 maintainer attention already.
I hear what you are saying, but I'm a bit hesitant to take style
suggestions at this point for fear of the situation where people ask
for changes back and forth across different versions. Unless any x86
maintainers want to chime in again? More responses below.
quoted
(Write=0,SavedDirty=1,Dirty=0):
- A modified, copy-on-write (COW) page. Previously when a typical
anonymous writable mapping was made COW via fork(), the kernel
would
mark it Write=0,Dirty=1. Now it will instead use the SavedDirty
bit.
This happens in copy_present_pte().
- A R/O page that has been COW'ed. The user page is in a R/O VMA,
and get_user_pages(FOLL_FORCE) needs a writable copy. The page
fault
handler creates a copy of the page and sets the new copy's PTE
as
Write=0 and SavedDirty=1.
- A shared shadow stack PTE. When a shadow stack page is being
shared
among processes (this happens at fork()), its PTE is made
Dirty=0, so
the next shadow stack access causes a fault, and the page is
duplicated and Dirty=1 is set again. This is the COW equivalent
for
shadow stack pages, even though it's copy-on-access rather than
copy-on-write.
(Write=0,SavedDirty=0,Dirty=1):
- A shadow stack PTE.
- A Cow PTE created when a processor without shadow stack support
set
Dirty=1.
There are six bits left available to software in the 64-bit PTE
after
consuming a bit for _PAGE_SAVED_DIRTY. No space is consumed in 32-
bit
kernels because shadow stacks are not enabled there.
Implement only the infrastructure for _PAGE_SAVED_DIRTY. Changes to
start
creating _PAGE_SAVED_DIRTY PTEs will follow once other pieces are
in place.
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Reviewed-by: Kees Cook <redacted>
Co-developed-by: Yu-cheng Yu <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
v6:
- Rename _PAGE_COW to _PAGE_SAVED_DIRTY (David Hildenbrand)
- Add _PAGE_SAVED_DIRTY to _PAGE_CHG_MASK
v5:
- Fix log, comments and whitespace (Boris)
- Remove capitalization on shadow stack (Boris)
v4:
- Teach pte_flags_need_flush() about _PAGE_COW bit
- Break apart patch for better bisectability
v3:
- Add comment around _PAGE_TABLE in response to comment
from (Andrew Cooper)
- Check for PSE in pmd_shstk (Andrew Cooper)
- Get to the point quicker in commit log (Andrew Cooper)
- Clarify and reorder commit log for why the PTE bit examples
have
multiple entries. Apply same changes for comment. (peterz)
- Fix comment that implied dirty bit for COW was a specific x86
thing
(peterz)
- Fix swapping of Write/Dirty (PeterZ)
---
arch/x86/include/asm/pgtable.h | 79
++++++++++++++++++++++++++++
arch/x86/include/asm/pgtable_types.h | 65 ++++++++++++++++++++---
arch/x86/include/asm/tlbflush.h | 3 +-
3 files changed, 138 insertions(+), 9 deletions(-)
diff --git a/arch/x86/include/asm/pgtable.h
b/arch/x86/include/asm/pgtable.h
index 2b423d697490..110e552eb602 100644
pteval_t clear)
return native_make_pte(v & ~clear);
}
+/*
+ * COW and other write protection operations can result in
Dirty=1,Write=0
+ * PTEs. But in the case of X86_FEATURE_USER_SHSTK, the software
SavedDirty bit
+ * is used, since the Dirty=1,Write=0 will result in the memory
being treated as
+ * shadow stack by the HW. So when creating dirty, write-protected
memory, a
+ * software bit is used _PAGE_BIT_SAVED_DIRTY. The following
functions
+ * pte_mksaveddirty() and pte_clear_saveddirty() take a
conventional dirty,
+ * write-protected PTE (Write=0,Dirty=1) and transition it to the
shadow stack
+ * compatible version. (Write=0,SavedDirty=1).
+ */
+static inline pte_t pte_mksaveddirty(pte_t pte)
+{
+ if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+ return pte;
+
+ pte = pte_clear_flags(pte, _PAGE_DIRTY);
+ return pte_set_flags(pte, _PAGE_SAVED_DIRTY);
+}
+
+static inline pte_t pte_clear_saveddirty(pte_t pte)
+{
+ /*
+ * _PAGE_SAVED_DIRTY is unnecessary on !X86_FEATURE_USER_SHSTK
kernels,
+ * since the HW dirty bit can be used without creating shadow
stack
+ * memory. See the _PAGE_SAVED_DIRTY definition for more
details.
+ */
+ if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+ return pte;
+
+ /*
+ * PTE is getting copied-on-write, so it will be dirtied
+ * if writable, or made shadow stack if shadow stack and
+ * being copied on access. Set the dirty bit for both
+ * cases.
+ */
+ pte = pte_set_flags(pte, _PAGE_DIRTY);
+ return pte_clear_flags(pte, _PAGE_SAVED_DIRTY);
+}
+
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
static inline int pte_uffd_wp(pte_t pte)
{
Nope, not "copy-on-write" :) It's more like "dirty bit when the hw-
dirty
bit cannot be used". Maybe simply drop the comment.
Oops, I missed this when I scrubbed _PAGE_COW. Thanks. Will fix.
quoted
+#else
+#define _PAGE_BIT_SAVED_DIRTY 0
+#endif
+
/* If _PAGE_BIT_PRESENT is clear, we use these: */
/* - if the user mapped it with PROT_NONE; pte_present gives true
*/
#define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL
@@ -117,6 +127,40 @@ #define _PAGE_SOFTW4 (_AT(pteval_t, 0)) #endif+/*+ * The hardware requires shadow stack to be read-only and Dirty.+ * _PAGE_SAVED_DIRTY is a software-only bit used to separate copy-
on-write
+ * PTEs from shadow stack PTEs:
I'd suggest phrasing this differently. COW is just one scenario
where
this can happen. Also, I don't think that the description of
"separation" is correct.
Something like the following maybe?
"
However, there are valid cases where the kernel might create read-
only
PTEs that are dirty (e.g., fork(), mprotect(), uffd-wp(), soft-dirty
tracking). In this case, the _PAGE_SAVED_DIRTY bit is used instead
of
the HW-dirty bit, to avoid creating a wrong "shadow stack" PTEs.
Such
PTEs have (Write=0,SavedDirty=1,Dirty=0) set.
Note that on processors without shadow stack support, the
_PAGE_SAVED_DIRTY remains unused.
"
The I would simply drop below (which is also too COW-specific I
think).
COW is the main situation where shadow stacks become read-only. So, as
an example it is nice in that COW covers all the scenarios discussed.
Again, do any x86 maintainers want to weigh in here?
quoted
+ *
+ * (Write=0,SavedDirty=1,Dirty=0):
+ * - A modified, copy-on-write (COW) page. Previously when a
typical
+ * anonymous writable mapping was made COW via fork(), the
kernel would
+ * mark it Write=0,Dirty=1. Now it will instead use the Cow
bit. This
+ * happens in copy_present_pte().
+ * - A R/O page that has been COW'ed. The user page is in a R/O
VMA,
+ * and get_user_pages(FOLL_FORCE) needs a writable copy. The
page fault
+ * handler creates a copy of the page and sets the new copy's
PTE as
+ * Write=0 and SavedDirty=1.
+ * - A shared shadow stack PTE. When a shadow stack page is being
shared
+ * among processes (this happens at fork()), its PTE is made
Dirty=0, so
+ * the next shadow stack access causes a fault, and the page is
+ * duplicated and Dirty=1 is set again. This is the COW
equivalent for
+ * shadow stack pages, even though it's copy-on-access rather
than
+ * copy-on-write.
+ *
+ * (Write=0,SavedDirty=0,Dirty=1):
+ * - A shadow stack PTE.
+ * - A Cow PTE created when a processor without shadow stack
support set
+ * Dirty=1.
+ */
From: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com> Date: 2023-02-20 22:08:19
On Mon, 2023-02-20 at 13:56 +0100, David Hildenbrand wrote:
On 18.02.23 22:14, Rick Edgecombe wrote:
quoted
From: Yu-cheng Yu <redacted>
The x86 Control-flow Enforcement Technology (CET) feature includes
a new
type of memory called shadow stack. This shadow stack memory has
some
unusual properties, which requires some core mm changes to function
properly.
A shadow stack PTE must be read-only and have _PAGE_DIRTY set.
However,
read-only and Dirty PTEs also exist for copy-on-write (COW) pages.
These
two cases are handled differently for page faults. Introduce
VM_SHADOW_STACK to track shadow stack VMAs.
I suggest simplifying and abstracting that description.
"New hardware extensions implement support for shadow stack memory,
such
as x86 Control-flow Enforcement Technology (CET). Let's add a new VM
flag to identify these areas, for example, to be used to properly
indicate shadow stack PTEs to the hardware."
Ah yea, that top blurb was added to all the non-x86 arch patches after
some feedback from Andrew Morton. He had said basically (in some more
colorful language) that the changelogs (at the time) were written
assuming the reader knows what a shadow stack is.
So it might be worth keeping a little more info in the log?
quoted
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Yu-cheng Yu <redacted>
Reviewed-by: Kirill A. Shutemov <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v6:
- Add comment about VM_SHADOW_STACK not being allowed with
VM_SHARED
(David Hildenbrand)
Might want to add some more meat to the patch description why that
is
the case.
Sure.
quoted
v3:
- Drop arch specific change in arch_vma_name(). The memory can
show as
anonymous (Kirill)
- Change CONFIG_ARCH_HAS_SHADOW_STACK to
CONFIG_X86_USER_SHADOW_STACK
in show_smap_vma_flags() (Boris)
---
Documentation/filesystems/proc.rst | 1 +
fs/proc/task_mmu.c | 3 +++
include/linux/mm.h | 8 ++++++++
3 files changed, 12 insertions(+)
diff --git a/Documentation/filesystems/proc.rst
b/Documentation/filesystems/proc.rst
index e224b6d5b642..115843e8cce3 100644
@@ -564,6 +564,7 @@ encoded manner. The codes are the following: mt arm64 MTE allocation tags are enabled um userfaultfd missing tracking uw userfaultfd wr-protect tracking+ ss shadow stack page == ======================================= Note that there is no guarantee that every flag and associated
@@ -315,11 +315,13 @@ extern unsigned int kobjsize(const void
*objp);
#define VM_HIGH_ARCH_BIT_2 34 /* bit only usable on 64-
bit architectures */
#define VM_HIGH_ARCH_BIT_3 35 /* bit only usable on 64-
bit architectures */
#define VM_HIGH_ARCH_BIT_4 36 /* bit only usable on 64-
bit architectures */
+#define VM_HIGH_ARCH_BIT_5 37 /* bit only usable on 64-bit
architectures */
#define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0)
#define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1)
#define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2)
#define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3)
#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4)
+#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5)
#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */
#ifdef CONFIG_ARCH_HAS_PKEYS
@@ -335,6 +337,12 @@ extern unsigned int kobjsize(const void
Should we abstract this to CONFIG_ARCH_USER_SHADOW_STACK, seeing
that
other architectures might similarly need it?
There was an ARCH_HAS_SHADOW_STACK but it got removed following this
discussion:
https://lore.kernel.org/lkml/d09e952d8ae696f687f0787dfeb7be7699c02913.camel@intel.com/
Now we have this new RFC for riscv as potentially a second
implementation. But it is still very early, and I'm not sure anyone
knows exactly what the similarities will be in a mature version. So I
think it would be better to refactor in an ARCH_HAS_SHADOW_STACK later
(and similar abstractions) once that series is more mature and we have
an idea of what pieces will be shared. I don't have a problem in
principle with an ARCH config, just don't think we should do it yet.
This only warns? Should it also enforce the state?
Hmm, you mean something like forcing Dirty=0 if Write=0?
The thing we are worried about here is some new x86 code that creates
Write=0,Dirty=1 PTEs directly because the developer is unaware or
forgot about shadow stack. The issue the warning actually caught was
kernel memory being marked Write=0,Dirty=1, which today is more about
consistency than any functional issue. But if some future hypothetical
code was creating a userspace PTE like this, and depending on the
memory being read-only, then the enforcement would be useful and
potentially save the day.
The downside is that it adds tricky logic into a low level helper that
shouldn't be required unless strange and wrong new code is added in the
future. And then it is still only useful if the warning doesn't catch
the issue in testing. And then there would be some slight risk that the
Dirty bit was expected to be there in some PTE without shadow stack
exposure, and a functional bug would be introduced.
I'm waffling here. I could be convinced either way. Hopefully that
helps characterize the dilemma at least.
From: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com> Date: 2023-02-20 22:38:38
On Sun, 2023-02-19 at 12:43 -0800, Kees Cook wrote:
On Sat, Feb 18, 2023 at 01:14:17PM -0800, Rick Edgecombe wrote:
quoted
The x86 Control-flow Enforcement Technology (CET) feature includes
a new
type of memory called shadow stack. This shadow stack memory has
some
unusual properties, which require some core mm changes to function
properly.
One of the properties is that the shadow stack pointer (SSP), which
is a
CPU register that points to the shadow stack like the stack pointer
points
to the stack, can't be pointing outside of the 32 bit address space
when
the CPU is executing in 32 bit mode. It is desirable to prevent
executing
in 32 bit mode when shadow stack is enabled because the kernel
can't easily
support 32 bit signals.
On x86 it is possible to transition to 32 bit mode without any
special
interaction with the kernel, by doing a "far call" to a 32 bit
segment.
So the shadow stack implementation can use this address space
behavior
as a feature, by enforcing that shadow stack memory is always
crated
outside of the 32 bit address space. This way userspace will
trigger a
general protection fault which will in turn trigger a segfault if
it
tries to transition to 32 bit mode with shadow stack enabled.
This provides a clean error generating border for the user if they
try
attempt to do 32 bit mode shadow stack, rather than leave the
kernel in a
half working state for userspace to be surprised by.
So to allow future shadow stack enabling patches to map shadow
stacks
out of the 32 bit address space, introduce MAP_ABOVE4G. The
behavior
is pretty much like MAP_32BIT, except that it has the opposite
address
range. The are a few differences though.
If both MAP_32BIT and MAP_ABOVE4G are provided, the kernel will use
the
MAP_ABOVE4G behavior. Like MAP_32BIT, MAP_ABOVE4G is ignored in a
32 bit
syscall.
Should the interface refuse to accept both set instead?
I guess that might be less surprising. But I think to do this would
either require adding logic to core mm or a new arch breakout. I
actually kind of wish there was an easy way to keep this flag from
being used from userspace and just be a kernel only thing. It is only
used internally in this series and there isn't any know use for
userspace.
From: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com> Date: 2023-02-20 22:44:25
On Mon, 2023-02-20 at 13:58 +0100, David Hildenbrand wrote:
On 18.02.23 22:14, Rick Edgecombe wrote:
quoted
From: Yu-cheng Yu <redacted>
The x86 Control-flow Enforcement Technology (CET) feature includes
a new
type of memory called shadow stack. This shadow stack memory has
some
unusual properties, which requires some core mm changes to function
properly.
Account shadow stack pages to stack memory.
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v3:
- Remove unneeded VM_SHADOW_STACK check in accountable_mapping()
(Kirill)
v2:
- Remove is_shadow_stack_mapping() and just change it to
directly bitwise
and VM_SHADOW_STACK.
Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.
Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for
is_shadow_stack_mapping().
---
mm/mmap.c | 2 ++
1 file changed, 2 insertions(+)
vm_flags_t flags, long npages)
mm->exec_vm += npages;
else if (is_stack_mapping(flags))
mm->stack_vm += npages;
+ else if (flags & VM_SHADOW_STACK)
+ mm->stack_vm += npages;
Why not modify is_stack_mapping() ?
It kind of sticks out a little in this conditional, but
is_stack_mapping() has this comment:
/*
* Stack area - automatically grows in one direction
*
* VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
* do_mmap() forbids all other combinations.
*/
Shadow stack don't grow, so it doesn't quite fit. There used to be an
is_shadow_stack_mapping(), but it was removed because all that was
needed (for the time being) was the simple bitwise AND:
https://lore.kernel.org/lkml/804adbac-61e6-0fd2-f726-5735fb290199@intel.com/
From: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com> Date: 2023-02-20 22:52:38
On Sun, 2023-02-19 at 12:41 -0800, Kees Cook wrote:
On Sat, Feb 18, 2023 at 01:14:12PM -0800, Rick Edgecombe wrote:
quoted
If a VMA has the VM_SHADOW_STACK flag, it is shadow stack memory.
So
when it is made writable with pte_mkwrite(), it should create
shadow
stack memory, not conventionally writable memory. Now that
pte_mkwrite()
takes a VMA, and places where shadow stack memory might be created
pass
one, pte_mkwrite() can know when it should do this.
So make pte_mkwrite() create shadow stack memory when the VMA has
the
VM_SHADOW_STACK flag. Do the same thing for pmd_mkwrite().
This requires referencing VM_SHADOW_STACK in these functions, which
are
currently defined in pgtable.h, however mm.h (where VM_SHADOW_STACK
is
located) can't be pulled in without causing problems for files that
reference pgtable.h. So also move pte/pmd_mkwrite() into pgtable.c,
where
they can safely reference VM_SHADOW_STACK.
Tested-by: Pengfei Xu <redacted>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Is there any realistic performance impact from making these not
inline
now?
Hmm, I can't say definitively. I would think in write protecting
operations, the big cost would not be the PTE setters. For mapping
things read-only from the beginning (user text, etc), I'm not sure. I
guess it gives the compiler less flexibility, but also gives it the
option to have one copy and so less text size overall for the kernel.
Are there any specific microbenchmarks we could run?
From: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com> Date: 2023-02-20 22:54:15
On Sun, 2023-02-19 at 19:42 -0800, Kees Cook wrote:
On Sat, Feb 18, 2023 at 01:13:52PM -0800, Rick Edgecombe wrote:
quoted
This series implements Shadow Stacks for userspace using x86's
Control-flow
Enforcement Technology (CET). CET consists of two related security
features:
shadow stacks and indirect branch tracking. This series implements
just the
shadow stack part of this feature, and just for userspace.
Okay, I've done some bare metal testing, and it all looks happy. The
selftest passes, and I can can see the stack address mismatch get
detected if I explicitly rewrite the saved function pointer on the
stack:
[INFO] Want normal flow
[INFO] Found 0x401890 @ 0x7fff47cf2ef8
[INFO] Normal execution flow
[INFO] Want to redirect
[INFO] Found 0x401890 @ 0x7fff47cf2ef8
[INFO] Hijacked execution flow
[INFO] Enabling shadow stack
[INFO] Want to redirect
[INFO] Found 0x401890 @ 0x7fff47cf2ef8
Segmentation fault (core dumped)
Tested-by: Kees Cook <redacted>
From: David Hildenbrand <hidden> Date: 2023-02-21 08:32:09
On 20.02.23 23:44, Edgecombe, Rick P wrote:
On Mon, 2023-02-20 at 13:58 +0100, David Hildenbrand wrote:
quoted
On 18.02.23 22:14, Rick Edgecombe wrote:
quoted
From: Yu-cheng Yu <redacted>
The x86 Control-flow Enforcement Technology (CET) feature includes
a new
type of memory called shadow stack. This shadow stack memory has
some
unusual properties, which requires some core mm changes to function
properly.
Account shadow stack pages to stack memory.
Reviewed-by: Kees Cook <redacted>
Tested-by: Pengfei Xu <redacted>
Tested-by: John Allen <john.allen@amd.com>
Signed-off-by: Yu-cheng Yu <redacted>
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Kees Cook <redacted>
---
v3:
- Remove unneeded VM_SHADOW_STACK check in accountable_mapping()
(Kirill)
v2:
- Remove is_shadow_stack_mapping() and just change it to
directly bitwise
and VM_SHADOW_STACK.
Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.
Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for
is_shadow_stack_mapping().
---
mm/mmap.c | 2 ++
1 file changed, 2 insertions(+)
vm_flags_t flags, long npages)
mm->exec_vm += npages;
else if (is_stack_mapping(flags))
mm->stack_vm += npages;
+ else if (flags & VM_SHADOW_STACK)
+ mm->stack_vm += npages;
Why not modify is_stack_mapping() ?
It kind of sticks out a little in this conditional, but
is_stack_mapping() has this comment:
/*
* Stack area - automatically grows in one direction
*
* VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
* do_mmap() forbids all other combinations.
*/
Shadow stack don't grow, so it doesn't quite fit. There used to be an
is_shadow_stack_mapping(), but it was removed because all that was
needed (for the time being) was the simple bitwise AND:
https://lore.kernel.org/lkml/804adbac-61e6-0fd2-f726-5735fb290199@intel.com/
As there is only a single user of is_stack_mapping(), I'd simply have
adjusted the doc of is_stack_mapping() to include shadow stacks.
--
Thanks,
David / dhildenb
From: David Hildenbrand <hidden> Date: 2023-02-21 08:37:29
On 20.02.23 23:08, Edgecombe, Rick P wrote:
On Mon, 2023-02-20 at 13:56 +0100, David Hildenbrand wrote:
quoted
On 18.02.23 22:14, Rick Edgecombe wrote:
quoted
From: Yu-cheng Yu <redacted>
The x86 Control-flow Enforcement Technology (CET) feature includes
a new
type of memory called shadow stack. This shadow stack memory has
some
unusual properties, which requires some core mm changes to function
properly.
A shadow stack PTE must be read-only and have _PAGE_DIRTY set.
However,
read-only and Dirty PTEs also exist for copy-on-write (COW) pages.
These
two cases are handled differently for page faults. Introduce
VM_SHADOW_STACK to track shadow stack VMAs.
I suggest simplifying and abstracting that description.
"New hardware extensions implement support for shadow stack memory,
such
as x86 Control-flow Enforcement Technology (CET). Let's add a new VM
flag to identify these areas, for example, to be used to properly
indicate shadow stack PTEs to the hardware."
Ah yea, that top blurb was added to all the non-x86 arch patches after
some feedback from Andrew Morton. He had said basically (in some more
colorful language) that the changelogs (at the time) were written
assuming the reader knows what a shadow stack is.
Okay. It's a bit repetitive, though.
Ideally, we'd just explain it in the cover letter in detail and
Andrews's script would include the cover letter in the first commit.
IIRC, that's what usually happens.
So it might be worth keeping a little more info in the log?
Copying the same paragraph into each commit is IMHO a bit repetitive.
But these are just my 2 cents.
[...]
quoted
Should we abstract this to CONFIG_ARCH_USER_SHADOW_STACK, seeing
that
other architectures might similarly need it?
There was an ARCH_HAS_SHADOW_STACK but it got removed following this
discussion:
https://lore.kernel.org/lkml/d09e952d8ae696f687f0787dfeb7be7699c02913.camel@intel.com/
Now we have this new RFC for riscv as potentially a second
implementation. But it is still very early, and I'm not sure anyone
knows exactly what the similarities will be in a mature version. So I
think it would be better to refactor in an ARCH_HAS_SHADOW_STACK later
(and similar abstractions) once that series is more mature and we have
an idea of what pieces will be shared. I don't have a problem in
principle with an ARCH config, just don't think we should do it yet.
Okay, easy to factor out later.
Acked-by: David Hildenbrand <redacted>
--
Thanks,
David / dhildenb