When a task does fork(), its shadow stack must be duplicated for
the child. However, the child may not actually use all pages of
of the copied shadow stack. This patch implements a flow that
is similar to copy-on-write of an anonymous page, but for shadow
stack memory. A shadow stack PTE needs to be RO and dirty. We
use this dirty bit requirement to effect the copying of shadow
stack pages.
In copy_one_pte(), we clear the dirty bit from the shadow stack
PTE. On the next shadow stack access to the PTE, a page fault
occurs. At that time, we then copy/re-use the page and fix the
PTE.
Signed-off-by: Yu-cheng Yu <redacted>
---
mm/memory.c | 30 ++++++++++++++++++++++++++++--
1 file changed, 28 insertions(+), 2 deletions(-)
@@ -2526,7 +2532,11 @@ static int wp_page_copy(struct vm_fault *vmf)}flush_cache_page(vma,vmf->address,pte_pfn(vmf->orig_pte));entry=mk_pte(new_page,vma->vm_page_prot);-entry=maybe_mkwrite(pte_mkdirty(entry),vma);+if(is_shstk_mapping(vma->vm_flags))+entry=pte_mkdirty_shstk(entry);+else+entry=pte_mkdirty(entry);+entry=maybe_mkwrite(entry,vma);/**Clearthepteentryandflushitfirst,beforeupdatingthe*ptewiththenewentry.Thiswillavoidaracecondition
@@ -3201,6 +3211,14 @@ static int do_anonymous_page(struct vm_fault *vmf)mem_cgroup_commit_charge(page,memcg,false,false);lru_cache_add_active_or_unevictable(page,vma);setpte:+/*+*Ifthisiswithinashadowstackmapping,mark+*thePTEdirty.Wedon'tusepte_mkdirty(),+*becausethePTEmusthave_PAGE_DIRTY_HWset.+*/+if(is_shstk_mapping(vma->vm_flags))+entry=pte_mkdirty_shstk(entry);+set_pte_at(vma->vm_mm,vmf->address,vmf->pte,entry);/* No need to invalidate - it was non-present before */
@@ -3983,6 +4001,14 @@ static int handle_pte_fault(struct vm_fault *vmf)entry=vmf->orig_pte;if(unlikely(!pte_same(*vmf->pte,entry)))gotounlock;++/*+*ShadowstackPTEsarecopy-on-access,sodo_wp_page()+*handlingonthemnomatterifwehavewritefaultornot.+*/+if(is_shstk_mapping(vmf->vma->vm_flags))+returndo_wp_page(vmf);+if(vmf->flags&FAULT_FLAG_WRITE){if(!pte_write(entry))returndo_wp_page(vmf);
This patch implements THP shadow stack memory copying in the same
way as the previous patch for regular PTE.
In copy_huge_pmd(), we clear the dirty bit from the PMD. On the
next shadow stack access to the PMD, a page fault occurs. At
that time, the page is copied/re-used and the PMD is fixed.
Signed-off-by: Yu-cheng Yu <redacted>
---
mm/huge_memory.c | 8 ++++++++
mm/memory.c | 8 +++++++-
2 files changed, 15 insertions(+), 1 deletion(-)
WRUSS is a new kernel-mode instruction but writes directly
to user shadow stack memory. This is used to construct
a return address on the shadow stack for the signal
handler.
This instruction can fault if the user shadow stack is
invalid shadow stack memory. In that case, the kernel does
fixup.
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/include/asm/special_insns.h | 45 +++++++++++++++++++
arch/x86/lib/x86-opcode-map.txt | 2 +-
arch/x86/mm/fault.c | 13 +++++-
tools/objtool/arch/x86/lib/x86-opcode-map.txt | 2 +-
4 files changed, 59 insertions(+), 3 deletions(-)
@@ -641,6 +641,17 @@ static int is_f00f_bug(struct pt_regs *regs, unsigned long address)return0;}+/*+*WRUSSisakernelinstrcutionandbutwritestouser+*shadowstackmemory.Whenafaultoccurs,both+*X86_PF_USERandX86_PF_SHSTKareset.+*/+staticintis_wruss(structpt_regs*regs,unsignedlongerror_code)+{+return(((error_code&(X86_PF_USER|X86_PF_SHSTK))==+(X86_PF_USER|X86_PF_SHSTK))&&!user_mode(regs));+}+staticvoidshow_fault_oops(structpt_regs*regs,unsignedlongerror_code,unsignedlongaddress)
@@ -848,7 +859,7 @@ __bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,structtask_struct*tsk=current;/* User mode accesses just cause a SIGSEGV */-if(error_code&X86_PF_USER){+if((error_code&X86_PF_USER)&&!is_wruss(regs,error_code)){/**It'spossibletohaveinterruptsoffhere:*/
@@ -241,3 +243,74 @@ int cet_setup_signal(bool ia32, unsigned long rstor_addr,set_shstk_ptr(ssp);return0;}++staticunsignedlongibt_mmap(unsignedlongaddr,unsignedlonglen)+{+structmm_struct*mm=current->mm;+unsignedlongpopulate;++down_write(&mm->mmap_sem);+addr=do_mmap(NULL,addr,len,PROT_READ|PROT_WRITE,+MAP_ANONYMOUS|MAP_PRIVATE,+VM_DONTDUMP,0,&populate,NULL);+up_write(&mm->mmap_sem);++if(populate)+mm_populate(addr,populate);++returnaddr;+}++intcet_setup_ibt(void)+{+u64r;++if(!cpu_feature_enabled(X86_FEATURE_IBT))+return-EOPNOTSUPP;++rdmsrl(MSR_IA32_U_CET,r);+r|=(MSR_IA32_CET_ENDBR_EN|MSR_IA32_CET_NO_TRACK_EN);+wrmsrl(MSR_IA32_U_CET,r);+current->thread.cet.ibt_enabled=1;+return0;+}++intcet_setup_ibt_bitmap(void)+{+u64r;+unsignedlongbitmap;+unsignedlongsize;++if(!cpu_feature_enabled(X86_FEATURE_IBT))+return-EOPNOTSUPP;++size=TASK_SIZE_MAX/PAGE_SIZE/BITS_PER_BYTE;+bitmap=ibt_mmap(0,size);++if(bitmap>=TASK_SIZE_MAX)+return-ENOMEM;++bitmap&=PAGE_MASK;++rdmsrl(MSR_IA32_U_CET,r);+r|=(MSR_IA32_CET_LEG_IW_EN|bitmap);+wrmsrl(MSR_IA32_U_CET,r);++current->thread.cet.ibt_bitmap_addr=bitmap;+current->thread.cet.ibt_bitmap_size=size;+return0;+}++voidcet_disable_ibt(void)+{+u64r;++if(!cpu_feature_enabled(X86_FEATURE_IBT))+return;++rdmsrl(MSR_IA32_U_CET,r);+r&=~(MSR_IA32_CET_ENDBR_EN|MSR_IA32_CET_LEG_IW_EN|+MSR_IA32_CET_NO_TRACK_EN);+wrmsrl(MSR_IA32_U_CET,r);+current->thread.cet.ibt_enabled=0;+}
arch_prctl(ARCH_CET_STATUS, unsigned long *addr)
Return CET feature status.
The parameter 'addr' is a pointer to a user buffer.
On returning to the caller, the kernel fills the following
information:
*addr = SHSTK/IBT status
*(addr + 1) = SHSTK base address
*(addr + 2) = SHSTK size
arch_prctl(ARCH_CET_DISABLE, unsigned long features)
Disable SHSTK and/or IBT specified in 'features'. Return -EPERM
if CET is locked out.
arch_prctl(ARCH_CET_LOCK)
Lock out CET feature.
arch_prctl(ARCH_CET_ALLOC_SHSTK, unsigned long *addr)
Allocate a new SHSTK.
The parameter 'addr' is a pointer to a user buffer and indicates
the desired SHSTK size to allocate. On returning to the caller
the buffer contains the address of the new SHSTK.
arch_prctl(ARCH_CET_LEGACY_BITMAP, unsigned long *addr)
Allocate an IBT legacy code bitmap if the current task does not
have one.
The parameter 'addr' is a pointer to a user buffer.
On returning to the caller, the kernel fills the following
information:
*addr = IBT bitmap base address
*(addr + 1) = IBT bitmap size
Signed-off-by: H.J. Lu <redacted>
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/include/asm/cet.h | 5 ++
arch/x86/include/uapi/asm/prctl.h | 6 ++
arch/x86/kernel/Makefile | 2 +-
arch/x86/kernel/cet.c | 26 ++++++
arch/x86/kernel/cet_prctl.c | 141 ++++++++++++++++++++++++++++++
arch/x86/kernel/elf.c | 4 +
arch/x86/kernel/process.c | 6 ++
7 files changed, 189 insertions(+), 1 deletion(-)
create mode 100644 arch/x86/kernel/cet_prctl.c
@@ -132,6 +132,32 @@ static unsigned long shstk_mmap(unsigned long addr, unsigned long len)returnaddr;}+intcet_alloc_shstk(unsignedlong*arg)+{+unsignedlonglen=*arg;+unsignedlongaddr;+unsignedlongtoken;+unsignedlongssp;++addr=shstk_mmap(0,len);+if(addr>=TASK_SIZE_MAX)+return-ENOMEM;++/* Restore token is 8 bytes and aligned to 8 bytes */+ssp=addr+len;+token=ssp;++if(!in_ia32_syscall())+token|=1;+ssp-=8;++if(write_user_shstk_64(ssp,token))+return-EINVAL;++*arg=addr;+return0;+}+intcet_setup_shstk(void){unsignedlongaddr,size;
@@ -795,6 +795,12 @@ long do_arch_prctl_common(struct task_struct *task, int option,returnget_cpuid_mode();caseARCH_SET_CPUID:returnset_cpuid_mode(task,cpuid_enabled);+caseARCH_CET_STATUS:+caseARCH_CET_DISABLE:+caseARCH_CET_LOCK:+caseARCH_CET_ALLOC_SHSTK:+caseARCH_CET_LEGACY_BITMAP:+returnprctl_cet(option,cpuid_enabled);}return-EINVAL;
The shadow stack for clone/fork is handled as the following:
(1) If ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM),
the kernel allocates (and frees on thread exit) a new SHSTK
for the child.
It is possible for the kernel to complete the clone syscall
and set the child's SHSTK pointer to NULL and let the child
thread allocate a SHSTK for itself. There are two issues
in this approach: It is not compatible with existing code
that does inline syscall and it cannot handle signals before
the child can successfully allocate a SHSTK.
(2) For (clone_flags & CLONE_VFORK), the child uses the existing
SHSTK.
(3) For all other cases, the SHSTK is copied/reused whenever the
parent or the child does a call/ret.
This patch handles cases (1) & (2). Case (3) is handled in
the SHSTK page fault patches.
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/include/asm/cet.h | 2 ++
arch/x86/include/asm/mmu_context.h | 3 +++
arch/x86/kernel/cet.c | 33 ++++++++++++++++++++++++++++++
arch/x86/kernel/process.c | 1 +
arch/x86/kernel/process_64.c | 7 +++++++
5 files changed, 46 insertions(+)
@@ -317,6 +317,13 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long sp,if(sp)childregs->sp=sp;+/* Allocate a new shadow stack for pthread */+if((clone_flags&(CLONE_VFORK|CLONE_VM))==CLONE_VM){+err=cet_setup_thread_shstk(p);+if(err)+gotoout;+}+err=-ENOMEM;if(unlikely(test_tsk_thread_flag(me,TIF_IO_BITMAP))){p->thread.io_bitmap_ptr=kmemdup(me->thread.io_bitmap_ptr,
From: "H.J. Lu" <redacted>
When Intel indirect branch tracking is enabled, functions in vDSO which
may be called indirectly must have endbr32 or endbr64 as the first
instruction. Compiler must support -fcf-protection=branch so that it
can be used to compile vDSO.
Signed-off-by: H.J. Lu <redacted>
---
arch/x86/entry/vdso/.gitignore | 4 ++++
arch/x86/entry/vdso/Makefile | 12 +++++++++++-
arch/x86/entry/vdso/vdso-layout.lds.S | 1 +
3 files changed, 16 insertions(+), 1 deletion(-)
When setting up a signal, the kernel creates a shadow stack
restore token at the current SHSTK address and then stores the
token's address in the signal frame, right after the FPU state.
Before restoring a signal, the kernel verifies and then uses the
restore token to set the SHSTK pointer.
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/ia32/ia32_signal.c | 13 +++
arch/x86/include/asm/cet.h | 5 ++
arch/x86/include/asm/sighandling.h | 5 ++
arch/x86/include/uapi/asm/sigcontext.h | 17 ++++
arch/x86/kernel/cet.c | 115 +++++++++++++++++++++++++
arch/x86/kernel/signal.c | 96 +++++++++++++++++++++
6 files changed, 251 insertions(+)
@@ -196,6 +196,23 @@ struct _xstate {/* New processor state extensions go here: */};+#ifdef __x86_64__+/*+*Sigcontextextension(structsc_ext)islocatedafter+*sigcontext->fpstate.Becausecurrentlyonlytheshadow+*stackpointerissavedthereandtheshadowstackdepends+*onXSAVES,wecanfindsc_extfromsigcontext->fpstate.+*+*The64-bitfpstatehasasizeoffpu_user_xstate_size,plus+*FP_XSTATE_MAGIC2_SIZEwhenXSAVE*isused.Thestructsc_ext+*islocatedattheendofsigcontext->fpstate,alignedto8.+*/+structsc_ext{+unsignedlongtotal_size;+unsignedlongssp;+};+#endif+/**The32-bitsignalframe:*/
@@ -49,6 +50,69 @@ static unsigned long get_shstk_addr(void)returnptr;}+/*+*Verifytherestoretokenattheaddressof'ssp'is+*validandthensetshadowstackpointeraccordingtothe+*token.+*/+staticintverify_rstor_token(boolia32,unsignedlongssp,+unsignedlong*new_ssp)+{+unsignedlongtoken;++*new_ssp=0;++if(!IS_ALIGNED(ssp,8))+return-EINVAL;++if(get_user(token,(unsignedlong__user*)ssp))+return-EFAULT;++/* Is 64-bit mode flag correct? */+if(ia32&&(token&3)!=0)+return-EINVAL;+elseif((token&3)!=1)+return-EINVAL;++token&=~(1UL);++if((!ia32&&!IS_ALIGNED(token,8))||!IS_ALIGNED(token,4))+return-EINVAL;++if((ALIGN_DOWN(token,8)-8)!=ssp)+return-EINVAL;++*new_ssp=token;+return0;+}++/*+*Createarestoretokenontheshadowstack.+*Atokenisalways8-byteandalignedto8.+*/+staticintcreate_rstor_token(boolia32,unsignedlongssp,+unsignedlong*new_ssp)+{+unsignedlongaddr;++*new_ssp=0;++if((!ia32&&!IS_ALIGNED(ssp,8))||!IS_ALIGNED(ssp,4))+return-EINVAL;++addr=ALIGN_DOWN(ssp,8)-8;++/* Is the token for 64-bit? */+if(!ia32)+ssp|=1;++if(write_user_shstk_64(addr,ssp))+return-EFAULT;++*new_ssp=addr;+return0;+}+staticunsignedlongshstk_mmap(unsignedlongaddr,unsignedlonglen){structmm_struct*mm=current->mm;
The user-mode indirect branch tracking support is done mostly by
GCC to insert ENDBR64/ENDBR32 instructions at branch targets.
The kernel provides CPUID enumeration, feature MSR setup and
the allocation of legacy bitmap.
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/Kconfig | 12 ++++++++++++
arch/x86/Makefile | 7 +++++++
2 files changed, 19 insertions(+)
@@ -164,6 +164,13 @@ ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER endifendif+# Check compiler ibt support+ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER+ ifeq ($(call cc-option-yn, -fcf-protection=branch), n)+$(errorCONFIG_X86_INTEL_BRANCH_TRACKING_USERnotsupportedbycompiler)+ endif+endif+## If the function graph tracer is used with mcount instead of fentry,# '-maccumulate-outgoing-args' is needed to prevent a GCC bug
VM_SHSTK indicates a shadow stack memory area.
A shadow stack PTE must be read-only and dirty. For non shadow
stack, we use a spare bit of the 64-bit PTE for dirty. The PTE
changes are in the next patch.
There is no more spare bit in the 32-bit PTE (except for PAE) and
the shadow stack is not implemented for the 32-bit kernel.
Signed-off-by: Yu-cheng Yu <redacted>
---
include/linux/mm.h | 8 ++++++++
mm/internal.h | 8 ++++++++
2 files changed, 16 insertions(+)
If a page fault is triggered by a shadow stack access (e.g.
call/ret) or shadow stack management instructions (e.g.
wrussq), then bit[6] of the page fault error code is set.
In access_error(), we check if a shadow stack page fault
is within a shadow stack memory area.
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/include/asm/traps.h | 2 ++
arch/x86/mm/fault.c | 11 +++++++++++
2 files changed, 13 insertions(+)
@@ -0,0 +1,250 @@+=========================================+Control Flow Enforcement Technology (CET)+=========================================++[1] Overview+============++Control Flow Enforcement Technology (CET) provides protection against+return/jump-oriented programing (ROP) attacks. It can be implemented+to protect both the kernel and applications. In the first phase,+only the user-mode protection is implemented for the 64-bit kernel.+Thirty-two bit applications are supported under the compatibility+mode.++CET includes shadow stack (SHSTK) and indirect branch tracking (IBT)+and they are enabled from two kernel configuration options:++ INTEL_X86_SHADOW_STACK_USER, and+ INTEL_X86_BRANCH_TRACKING_USER.++To build a CET-enabled kernel, Binutils v2.30 and GCC v8.1 or later+are required. To build a CET-enabled application, GLIBC v2.29 or+later is also requried.++There are two command-line options for disabling CET features:++ no_cet_shstk - disables SHSTK, and+ no_cet_ibt - disables IBT.++At run time, /proc/cpuinfo shows the availability of SHSTK and IBT.++[2] CET assembly instructions+=============================++RDSSP %r+ Read the SHSTK pointer into %r.++INCSSP %r+ Unwind (increment) the SHSTK pointer (0 ~ 255) steps as indicated+ in the operand register. The GLIBC longjmp uses INCSSP to unwind+ the SHSTK until that matches the program stack. When it is+ necessary to unwind beyond 255 steps, longjmp divides and repeats+ the process.++RSTORSSP (%r)+ Switch to the SHSTK indicated in the 'restore token' pointed by+ the operand register and replace the 'restore token' with a new+ token to be saved (with SAVEPREVSSP) for the outgoing SHSTK.++ Before RSTORSSP++ Incoming SHSTK Current/Outgoing SHSTK++ |----------------------| |----------------------|+ addr=x | | ssp-> | |+ |----------------------| |----------------------|+ (%r)-> | rstor_token=(x|Lg) | addr=y-8 | |+ |----------------------| |----------------------|++ After RSTORSSP++ |----------------------| |----------------------|+ ssp-> | | | |+ |----------------------| |----------------------|+ | rstor_token=(y|Bz|Lg)| addr=y-8 | |+ |----------------------| |----------------------|++ note:+ 1. Only valid addresses and restore tokens can be on the+ user-mode SHSTK.+ 2. A token is always of type u64 and must align to u64.+ 3. The incoming SHSTK pointer in a rstor_token must point to+ immediately above the token.+ 4. 'Lg' is bit[0] of a rstor_token indicating a 64-bit SHSTK.+ 5. 'Bz' is bit[1] of a rstor_token indicating the token is to+ be used only for the next SAVEPREVSSP and invalid for the+ RSTORSSP.++SAVEPREVSSP+ Store the SHSTK 'restore token' pointed by+ (current_SHSTK_pointer + 8).++ After SAVEPREVSSP++ |----------------------| |----------------------|+ ssp-> | | | |+ |----------------------| |----------------------|+ | rstor_token=(y|Bz|Lg)| addr=y-8 | rstor_token(y|Lg) |+ |----------------------| |----------------------|++WRUSS %r0, (%r1)+ Write the value in %r0 to the SHSTK address pointed by (%r1).+ This is a kernel-mode only instruction.++ENDBR+ The compiler inserts an ENDBR at all valid branch targets. Any+ CALL/JMP to a target without an ENDBR triggers a control+ protection fault.++[3] Application Enabling+========================++An application's CET capability is marked in its ELF header and can+be verified from the following command output, in the+NT_GNU_PROPERTY_TYPE_0 field:++ readelf -n <application>++If an application supports CET and is statically linked, it will run+with CET protection. If the application needs any shared libraries,+the loader checks all dependencies and enables CET only when all+requirements are met.++[4] Legacy Libraries+====================++GLIBC provides a few tunables for backward compatibility.++GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT+ Turn off SHSTK/IBT for the current shell.++GLIBC_TUNABLES=glibc.tune.x86_shstk=<on, permissive>+ This controls how dlopen() handles SHSTK legacy libraries:+ on: continue with SHSTK enabled;+ permissive: continue with SHSTK off.++[5] CET system calls+====================++The following arch_prctl() system calls are added for CET:++arch_prctl(ARCH_CET_STATUS, unsigned long *addr)+ Return CET feature status.++ The parameter 'addr' is a pointer to a user buffer.+ On returning to the caller, the kernel fills the following+ information:++ *addr = SHSTK/IBT status+ *(addr + 1) = SHSTK base address+ *(addr + 2) = SHSTK size++arch_prctl(ARCH_CET_DISABLE, unsigned long features)+ Disable SHSTK and/or IBT specified in 'features'. Return -EPERM+ if CET is locked out.++arch_prctl(ARCH_CET_LOCK)+ Lock out CET feature.++arch_prctl(ARCH_CET_ALLOC_SHSTK, unsigned long *addr)+ Allocate a new SHSTK.++ The parameter 'addr' is a pointer to a user buffer and indicates+ the desired SHSTK size to allocate. On returning to the caller+ the buffer contains the address of the new SHSTK.++arch_prctl(ARCH_CET_LEGACY_BITMAP, unsigned long *addr)+ Allocate an IBT legacy code bitmap if the current task does not+ have one.++ The parameter 'addr' is a pointer to a user buffer.+ On returning to the caller, the kernel fills the following+ information:++ *addr = IBT bitmap base address+ *(addr + 1) = IBT bitmap size++[6] The implementation of the SHSTK+===================================++SHSTK size+----------++A task's SHSTK is allocated from memory to a fixed size that can+support 32 KB nested function calls; that is 256 KB for a 64-bit+application and 128 KB for a 32-bit application. The system admin+can change the default size.++Signal+------++The main program and its signal handlers use the same SHSTK. Because+the SHSTK stores only return addresses, we can estimate a large+enough SHSTK to cover the condition that both the program stack and+the sigaltstack run out.++The kernel creates a restore token at the SHSTK restoring address and+verifies that token when restoring from the signal handler.++Fork+----++The SHSTK's vma has VM_SHSTK flag set; its PTEs are required to be+read-only and dirty. When a SHSTK PTE is not present, RO, and dirty,+a SHSTK access triggers a page fault with an additional SHSTK bit set+in the page fault error code.++When a task forks a child, its SHSTK PTEs are copied and both the+parent's and the child's SHSTK PTEs are cleared of the dirty bit.+Upon the next SHSTK access, the resulting SHSTK page fault is handled+by page copy/re-use.++When a pthread child is created, the kernel allocates a new SHSTK for+the new thread.++Setjmp/Longjmp+--------------++Longjmp unwinds SHSTK until it matches the program stack.++Ucontext+--------++In GLIBC, getcontext/setcontext is implemented in similar way as+setjmp/longjmp.++When makecontext creates a new ucontext, a new SHSTK is allocated for+that context with ARCH_CET_ALLOC_SHSTK the syscall. The kernel+creates a restore token at the top of the new SHSTK and the user-mode+code switches to the new SHSTK with the RSTORSSP instruction.++[7] The management of read-only & dirty PTEs for SHSTK+======================================================++A RO and dirty PTE exists in the following cases:++(a) A page is modified and then shared with a fork()'ed child;+(b) A R/O page that has been COW'ed;+(c) A SHSTK page.++The processor only checks the dirty bit for (c). To prevent the use+of non-SHSTK memory as SHSTK, we use a spare bit of the 64-bit PTE as+DIRTY_SW for (a) and (b) above. This results to the following PTE+settings:++Modified PTE: (R/W + DIRTY_HW)+Modified and shared PTE: (R/O + DIRTY_SW)+R/O PTE, COW'ed: (R/O + DIRTY_SW)+SHSTK PTE: (R/O + DIRTY_HW)+SHSTK PTE, COW'ed: (R/O + DIRTY_HW)+SHSTK PTE, shared: (R/O + DIRTY_SW)++Note that DIRTY_SW is only used in R/O PTEs but not R/W PTEs.++[8] The implementation of IBT+=============================++The kernel provides IBT support in mmap() of the legacy code bit map.+However, the management of the bitmap is done in the GLIBC or the+application.
Introduce Kconfig option X86_INTEL_SHADOW_STACK_USER.
An application has shadow stack protection when all the following are
true:
(1) The kernel has X86_INTEL_SHADOW_STACK_USER enabled,
(2) The running processor supports the shadow stack,
(3) The application is built with shadow stack enabled tools & libs
and, and at runtime, all dependent shared libs can support shadow
stack.
If this kernel config option is enabled, but (2) or (3) above is not
true, the application runs without the shadow stack protection.
Existing legacy applications will continue to work without the shadow
stack protection.
The user-mode shadow stack protection is only implemented for the
64-bit kernel. Thirty-two bit applications are supported under the
compatibility mode.
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/Kconfig | 24 ++++++++++++++++++++++++
arch/x86/Makefile | 7 +++++++
2 files changed, 31 insertions(+)
@@ -157,6 +157,13 @@ ifdef CONFIG_X86_X32endifexportCONFIG_X86_X32_ABI+# Check assembler shadow stack suppot+ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER+ ifeq ($(call as-instr, saveprevssp, y),)+$(errorCONFIG_X86_INTEL_SHADOW_STACK_USERnotsupportedbytheassembler)+ endif+endif+## If the function graph tracer is used with mcount instead of fentry,# '-maccumulate-outgoing-args' is needed to prevent a GCC bug
A control protection exception is triggered when a control flow transfer
attempt violated shadow stack or indirect branch tracking constraints.
For example, the return address for a RET instruction differs from the
safe copy on the shadow stack; or a JMP instruction arrives at a non-
ENDBR instruction.
The control protection exception handler works in a similar way as the
general protection fault handler.
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/entry/entry_64.S | 2 +-
arch/x86/include/asm/traps.h | 3 ++
arch/x86/kernel/idt.c | 4 +++
arch/x86/kernel/traps.c | 58 ++++++++++++++++++++++++++++++++++++
4 files changed, 66 insertions(+), 1 deletion(-)
@@ -578,6 +578,64 @@ do_general_protection(struct pt_regs *regs, long error_code)}NOKPROBE_SYMBOL(do_general_protection);+staticconstchar*control_protection_err[]=+{+"unknown",+"near-ret",+"far-ret/iret",+"endbranch",+"rstorssp",+"setssbsy",+};++/*+*Whenacontrolprotectionexceptionoccurs,sendasignal+*totheresponsibleapplication.Currently,control+*protectionisonlyenabledfortheusermode.This+*exceptionshouldnotcomefromthekernelmode.+*/+dotraplinkagevoid+do_control_protection(structpt_regs*regs,longerror_code)+{+structtask_struct*tsk;++RCU_LOCKDEP_WARN(!rcu_is_watching(),"entry code didn't wake RCU");+if(notify_die(DIE_TRAP,"control protection fault",regs,+error_code,X86_TRAP_CP,SIGSEGV)==NOTIFY_STOP)+return;+cond_local_irq_enable(regs);++if(!user_mode(regs))+die("kernel control protection fault",regs,error_code);++if(!static_cpu_has(X86_FEATURE_SHSTK)&&+!static_cpu_has(X86_FEATURE_IBT))+WARN_ONCE(1,"CET is disabled but got control "+"protection fault\n");++tsk=current;+tsk->thread.error_code=error_code;+tsk->thread.trap_nr=X86_TRAP_CP;++if(show_unhandled_signals&&unhandled_signal(tsk,SIGSEGV)&&+printk_ratelimit()){+unsignedintmax_err;++max_err=ARRAY_SIZE(control_protection_err)-1;+if((error_code<0)||(error_code>max_err))+error_code=0;+pr_info("%s[%d] control protection ip:%lx sp:%lx error:%lx(%s)",+tsk->comm,task_pid_nr(tsk),+regs->ip,regs->sp,error_code,+control_protection_err[error_code]);+print_vma_addr(" in ",regs->ip);+pr_cont("\n");+}++force_sig_info(SIGSEGV,SEND_SIG_PRIV,tsk);+}+NOKPROBE_SYMBOL(do_control_protection);+dotraplinkagevoidnotracedo_int3(structpt_regs*regs,longerror_code){#ifdef CONFIG_DYNAMIC_FTRACE
A RO and dirty PTE exists in the following cases:
(a) A page is modified and then shared with a fork()'ed child;
(b) A R/O page that has been COW'ed;
(c) A SHSTK page.
The processor does not read the dirty bit for (a) and (b), but
checks the dirty bit for (c). To prevent the use of non-SHSTK
memory as SHSTK, we introduce a spare bit of the 64-bit PTE as
_PAGE_BIT_DIRTY_SW and use that for (a) and (b). This results
to the following possible PTE settings:
Modified PTE: (R/W + DIRTY_HW)
Modified and shared PTE: (R/O + DIRTY_SW)
R/O PTE COW'ed: (R/O + DIRTY_SW)
SHSTK PTE: (R/O + DIRTY_HW)
SHSTK PTE COW'ed: (R/O + DIRTY_HW)
SHSTK PTE shared: (R/O + DIRTY_SW)
Note that _PAGE_BIT_DRITY_SW is only used in R/O PTEs but
not R/W PTEs.
When this patch is applied, there are six free bits left in
the 64-bit PTE. There is no more free bit in the 32-bit
PTE (except for PAE) and shadow stack is not implemented
for the 32-bit kernel.
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/include/asm/pgtable.h | 109 ++++++++++++++++++++++++---
arch/x86/include/asm/pgtable_types.h | 14 +++-
include/asm-generic/pgtable.h | 21 ++++++
3 files changed, 132 insertions(+), 12 deletions(-)
@@ -23,6 +23,7 @@#define _PAGE_BIT_SOFTW2 10 /* " */#define _PAGE_BIT_SOFTW3 11 /* " */#define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */+#define _PAGE_BIT_SOFTW5 57 /* available for programmer */#define _PAGE_BIT_SOFTW4 58 /* available for programmer */#define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */#define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */
@@ -34,6 +35,7 @@#define _PAGE_BIT_CPA_TEST _PAGE_BIT_SOFTW1#define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */#define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4+#define _PAGE_BIT_DIRTY_SW _PAGE_BIT_SOFTW5 /* was written to *//* If _PAGE_BIT_PRESENT is clear, we use these: *//* - if the user mapped it with PROT_NONE; pte_present gives true */
XSAVES saves both system and user states. The Linux kernel
currently does not save/restore any system states. This patch
creates the framework for supporting system states.
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/include/asm/fpu/internal.h | 3 +-
arch/x86/include/asm/fpu/xstate.h | 9 ++-
arch/x86/kernel/fpu/core.c | 7 +-
arch/x86/kernel/fpu/init.c | 10 ---
arch/x86/kernel/fpu/xstate.c | 112 +++++++++++++++++-----------
5 files changed, 80 insertions(+), 61 deletions(-)
@@ -19,10 +19,10 @@#define XSAVE_YMM_SIZE 256#define XSAVE_YMM_OFFSET (XSAVE_HDR_SIZE + XSAVE_HDR_OFFSET)-/* System features */-#define XFEATURE_MASK_SYSTEM (XFEATURE_MASK_PT)--/* All currently supported features */+/*+*SUPPORTED_XFEATURES_MASKindicatesallfeatures+*implementedinandsupportedbythekernel.+*/#define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \XFEATURE_MASK_SSE|\XFEATURE_MASK_YMM|\
@@ -219,30 +222,31 @@ void fpstate_sanitize_xstate(struct fpu *fpu)*/voidfpu__init_cpu_xstate(void){-if(!boot_cpu_has(X86_FEATURE_XSAVE)||!xfeatures_mask_user)+if(!boot_cpu_has(X86_FEATURE_XSAVE)||!xfeatures_mask_all)return;++cr4_set_bits(X86_CR4_OSXSAVE);+/*-*MakeitclearthatXSAVESsystemstatesarenotyet-*implementedshouldanyoneexpectittoworkbychanging-*bitsinXFEATURE_MASK_*macrosandXCR0.+*XCR_XFEATURE_ENABLED_MASKsetsthefeaturesthataremanaged+*byXSAVE{C,OPT}andXRSTOR.OnlyXSAVEuserstatescanbe+*sethere.*/-WARN_ONCE((xfeatures_mask_user&XFEATURE_MASK_SYSTEM),-"x86/fpu: XSAVES system states are not yet implemented.\n");+xsetbv(XCR_XFEATURE_ENABLED_MASK,+xfeatures_mask_user);-xfeatures_mask_user&=~XFEATURE_MASK_SYSTEM;--cr4_set_bits(X86_CR4_OSXSAVE);-xsetbv(XCR_XFEATURE_ENABLED_MASK,xfeatures_mask_user);+/*+*MSR_IA32_XSSsetswhichXSAVESsystemstatestobemanagedby+*XSAVES.OnlyXSAVESsystemstatescanbesethere.+*/+if(boot_cpu_has(X86_FEATURE_XSAVES))+wrmsrl(MSR_IA32_XSS,+xfeatures_mask_all&~xfeatures_mask_user);}-/*-*Notethatinthefuturewewilllikelyneedapairof-*functionshere:oneforuserxstatesandtheotherfor-*systemxstates.Fornow,theyarethesame.-*/staticintxfeature_enabled(enumxfeaturexfeature){-return!!(xfeatures_mask_user&BIT_ULL(xfeature));+return!!(xfeatures_mask_all&BIT_ULL(xfeature));}/*
@@ -348,7 +352,7 @@ static int xfeature_is_aligned(int xfeature_nr)*/staticvoid__initsetup_xstate_comp(void){-unsignedintxstate_comp_sizes[sizeof(xfeatures_mask_user)*8];+unsignedintxstate_comp_sizes[sizeof(xfeatures_mask_all)*8];inti;/*
@@ -441,11 +445,10 @@ static int xfeature_uncompacted_offset(int xfeature_nr)u32eax,ebx,ecx,edx;/*-*OnlyXSAVESsupportssystemstatesanditusescompacted-*format.Checkingasystemstate'suncompactedoffsetis-*anerror.+*Checkingasystemorunsupportedstate'suncompactedoffset+*isanerror.*/-if(XFEATURE_MASK_SYSTEM&(1<<xfeature_nr)){+if(~xfeatures_mask_user&BIT_ULL(xfeature_nr)){WARN_ONCE(1,"No fixed offset for xstate %d\n",xfeature_nr);return-1;}
@@ -482,7 +485,7 @@ int using_compacted_format(void)intvalidate_xstate_header(conststructxstate_header*hdr){/* No unknown or system features may be set */-if(hdr->xfeatures&(~xfeatures_mask_user|XFEATURE_MASK_SYSTEM))+if(hdr->xfeatures&~xfeatures_mask_user)return-EINVAL;/* Userspace must use the uncompacted format */
@@ -760,10 +777,11 @@ void __init fpu__init_system_xstate(void)*/for(i=0;i<ARRAY_SIZE(xsave_cpuid_features);i++){if(!boot_cpu_has(xsave_cpuid_features[i]))-xfeatures_mask_user&=~BIT_ULL(i);+xfeatures_mask_all&=~BIT_ULL(i);}-xfeatures_mask_user&=fpu__get_supported_xfeatures_mask();+xfeatures_mask_all&=SUPPORTED_XFEATURES_MASK;+xfeatures_mask_user=xfeatures_mask_all&cpu_user_xfeatures_mask;/* Enable xstate instructions to be able to continue with initialization: */fpu__init_cpu_xstate();
To support XSAVES system states, change some names to distinguish
user and system states.
Change:
supervisor to system
copy_init_fpstate_to_fpregs() to copy_init_user_fpstate_to_fpregs()
xfeatures_mask to xfeatures_mask_user
XCNTXT_MASK to SUPPORTED_XFEATURES_MASK (states supported)
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/include/asm/fpu/internal.h | 5 +-
arch/x86/include/asm/fpu/xstate.h | 24 ++++----
arch/x86/kernel/fpu/core.c | 4 +-
arch/x86/kernel/fpu/init.c | 2 +-
arch/x86/kernel/fpu/signal.c | 6 +-
arch/x86/kernel/fpu/xstate.c | 88 +++++++++++++++--------------
6 files changed, 66 insertions(+), 63 deletions(-)
@@ -219,20 +219,20 @@ void fpstate_sanitize_xstate(struct fpu *fpu)*/voidfpu__init_cpu_xstate(void){-if(!boot_cpu_has(X86_FEATURE_XSAVE)||!xfeatures_mask)+if(!boot_cpu_has(X86_FEATURE_XSAVE)||!xfeatures_mask_user)return;/*-*MakeitclearthatXSAVESsupervisorstatesarenotyet+*MakeitclearthatXSAVESsystemstatesarenotyet*implementedshouldanyoneexpectittoworkbychanging*bitsinXFEATURE_MASK_*macrosandXCR0.*/-WARN_ONCE((xfeatures_mask&XFEATURE_MASK_SUPERVISOR),-"x86/fpu: XSAVES supervisor states are not yet implemented.\n");+WARN_ONCE((xfeatures_mask_user&XFEATURE_MASK_SYSTEM),+"x86/fpu: XSAVES system states are not yet implemented.\n");-xfeatures_mask&=~XFEATURE_MASK_SUPERVISOR;+xfeatures_mask_user&=~XFEATURE_MASK_SYSTEM;cr4_set_bits(X86_CR4_OSXSAVE);-xsetbv(XCR_XFEATURE_ENABLED_MASK,xfeatures_mask);+xsetbv(XCR_XFEATURE_ENABLED_MASK,xfeatures_mask_user);}/*
@@ -440,11 +441,11 @@ static int xfeature_uncompacted_offset(int xfeature_nr)u32eax,ebx,ecx,edx;/*-*OnlyXSAVESsupportssupervisorstatesanditusescompacted-*format.Checkingasupervisorstate'suncompactedoffsetis+*OnlyXSAVESsupportssystemstatesanditusescompacted+*format.Checkingasystemstate'suncompactedoffsetis*anerror.*/-if(XFEATURE_MASK_SUPERVISOR&(1<<xfeature_nr)){+if(XFEATURE_MASK_SYSTEM&(1<<xfeature_nr)){WARN_ONCE(1,"No fixed offset for xstate %d\n",xfeature_nr);return-1;}
@@ -465,7 +466,7 @@ static int xfeature_size(int xfeature_nr)/**'XSAVES'impliestwodifferentthings:-*1.savingofsupervisor/systemstate+*1.savingofsystemstate*2.usingthecompactedformat**Usethisfunctionwhendealingwiththecompactedformatso
@@ -480,8 +481,8 @@ int using_compacted_format(void)/* Validate an xstate header supplied by userspace (ptrace or sigreturn) */intvalidate_xstate_header(conststructxstate_header*hdr){-/* No unknown or supervisor features may be set */-if(hdr->xfeatures&(~xfeatures_mask|XFEATURE_MASK_SUPERVISOR))+/* No unknown or system features may be set */+if(hdr->xfeatures&(~xfeatures_mask_user|XFEATURE_MASK_SYSTEM))return-EINVAL;/* Userspace must use the uncompacted format */
@@ -588,11 +589,11 @@ static void do_extra_xstate_size_checks(void)check_xstate_against_struct(i);/*-*Supervisorstatecomponentscanbemanagedonlyby+*Systemstatecomponentscanbemanagedonlyby*XSAVES,whichiscompacted-formatonly.*/if(!using_compacted_format())-XSTATE_WARN_ON(xfeature_is_supervisor(i));+XSTATE_WARN_ON(xfeature_is_system(i));/* Align from the end of the previous feature */if(xfeature_is_aligned(i))
@@ -706,7 +707,7 @@ static int init_xstate_size(void)*/staticvoidfpu__init_disable_system_xstate(void){-xfeatures_mask=0;+xfeatures_mask_user=0;cr4_clear_bits(X86_CR4_OSXSAVE);fpu__xstate_clear_all_cpu_caps();}
@@ -742,15 +743,15 @@ void __init fpu__init_system_xstate(void)}cpuid_count(XSTATE_CPUID,0,&eax,&ebx,&ecx,&edx);-xfeatures_mask=eax+((u64)edx<<32);+xfeatures_mask_user=eax+((u64)edx<<32);-if((xfeatures_mask&XFEATURE_MASK_FPSSE)!=XFEATURE_MASK_FPSSE){+if((xfeatures_mask_user&XFEATURE_MASK_FPSSE)!=XFEATURE_MASK_FPSSE){/**Thisindicatesthatsomethingreallyunexpectedhappened*withtheenumeration.DisableXSAVEandtrytocontinue*bootingwithoutit.ThisistooearlytoBUG().*/-pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n",xfeatures_mask);+pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n",xfeatures_mask_user);gotoout_disable;}
@@ -759,10 +760,10 @@ void __init fpu__init_system_xstate(void)*/for(i=0;i<ARRAY_SIZE(xsave_cpuid_features);i++){if(!boot_cpu_has(xsave_cpuid_features[i]))-xfeatures_mask&=~BIT(i);+xfeatures_mask_user&=~BIT_ULL(i);}-xfeatures_mask&=fpu__get_supported_xfeatures_mask();+xfeatures_mask_user&=fpu__get_supported_xfeatures_mask();/* Enable xstate instructions to be able to continue with initialization: */fpu__init_cpu_xstate();
We are going to create _PAGE_DIRTY_SW for non-hardware, memory
management purposes. Rename _PAGE_DIRTY to _PAGE_DIRTY_HW and
_PAGE_BIT_DIRTY to _PAGE_BIT_DIRTY_HW to make these PTE dirty
bits more clear. There are no functional changes in this
patch.
Signed-off-by: Yu-cheng Yu <redacted>
---
arch/x86/include/asm/pgtable.h | 6 +++---
arch/x86/include/asm/pgtable_types.h | 17 +++++++++--------
arch/x86/kernel/relocate_kernel_64.S | 2 +-
arch/x86/kvm/vmx.c | 2 +-
4 files changed, 14 insertions(+), 13 deletions(-)
@@ -5467,7 +5467,7 @@ static int init_rmode_identity_map(struct kvm *kvm)/* Set up identity-mapping pagetable for EPT in real mode */for(i=0;i<PT32_ENT_PER_PAGE;i++){tmp=(i<<22)+(_PAGE_PRESENT|_PAGE_RW|_PAGE_USER|-_PAGE_ACCESSED|_PAGE_DIRTY|_PAGE_PSE);+_PAGE_ACCESSED|_PAGE_DIRTY_HW|_PAGE_PSE);r=kvm_write_guest_page(kvm,identity_map_pfn,&tmp,i*sizeof(tmp),sizeof(tmp));if(r<0)
From: Dave Hansen <dave.hansen@linux.intel.com> Date: 2018-07-10 22:44:52
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
+ /*
+ * On platforms before CET, other threads could race to
+ * create a RO and _PAGE_DIRTY_HW PMD again. However,
+ * on CET platforms, this is safe without a TLB flush.
+ */
If I didn't work for Intel, I'd wonder what the heck CET is and what the
heck it has to do with _PAGE_DIRTY_HW. I think we need a better comment
than this. How about:
Some processors can _start_ a write, but end up seeing
a read-only PTE by the time they get to getting the
Dirty bit. In this case, they will set the Dirty bit,
leaving a read-only, Dirty PTE which looks like a Shadow
Stack PTE.
However, this behavior has been improved and will *not* occur on
processors supporting Shadow Stacks. Without this guarantee, a
transition to a non-present PTE and flush the TLB would be
needed.
Could we document this bit better?
Is this a fault where the *processor* thought it should be a shadow
stack fault? Or is it also set on faults to valid shadow stack PTEs
that just happen to fault for other reasons, say protection keys?
Do we want to lift this hunk of code and put it elsewhere? Maybe:
entry = pte_set_vma_features(entry, vma);
and then:
pte_t pte_set_vma_features(pte_t entry, struct vm_area_struct)
{
/*
* Shadow stack PTEs are always dirty and always
* writable. They have a different encoding for
* this than normal PTEs, though.
*/
if (is_shstk_mapping(vma->vm_flags))
entry = pte_mkdirty_shstk(entry);
else
entry = pte_mkdirty(entry);
entry = maybe_mkwrite(entry, vma);
return entry;
}
quoted hunk
/*
* Clear the pte entry and flush it first, before updating the
* pte with the new entry. This will avoid a race condition
@@ -3201,6 +3211,14 @@ static int do_anonymous_page(struct vm_fault *vmf) mem_cgroup_commit_charge(page, memcg, false, false); lru_cache_add_active_or_unevictable(page, vma); setpte:+ /*+ * If this is within a shadow stack mapping, mark+ * the PTE dirty. We don't use pte_mkdirty(),+ * because the PTE must have _PAGE_DIRTY_HW set.+ */+ if (is_shstk_mapping(vma->vm_flags))+ entry = pte_mkdirty_shstk(entry);+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
I'm not sure this is the right spot to do this.
The other code does pte_mkdirty_shstk() near where we do the
pte_mkwrite(). Why not here? I think you might have picked this
because it's a common path used by both allocated pages and zero pages.
But, we can't have the zero pages in shadow stack areas since they can't
be read-only. I think you need to move this up. Can you even
consolidate it with the other two pte_mkdirt_shstk() call sites?
quoted hunk
/* No need to invalidate - it was non-present before */
@@ -3983,6 +4001,14 @@ static int handle_pte_fault(struct vm_fault *vmf) entry = vmf->orig_pte; if (unlikely(!pte_same(*vmf->pte, entry))) goto unlock;++ /*+ * Shadow stack PTEs are copy-on-access, so do_wp_page()+ * handling on them no matter if we have write fault or not.+ */
I'd say this differently:
Shadow stack PTEs can not be read-only and because of that can
not have traditional copy-on-write semantics. This essentially
performs a copy-on-write operation, but on *any* access, not
just actual writes.
This pattern is repeated enough that it makes me wonder if we should
just be doing the shadowstack PTE creation in mk_huge_pmd() itself.
Or, should we just be setting the shadowstack pte bit combination in
vma->vm_page_prot so we don't have to go set it explicitly every time?
From: Nadav Amit <hidden> Date: 2018-07-10 23:23:15
at 6:44 PM, Dave Hansen [off-list ref] wrote:
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
quoted
+ /*
+ * On platforms before CET, other threads could race to
+ * create a RO and _PAGE_DIRTY_HW PMD again. However,
+ * on CET platforms, this is safe without a TLB flush.
+ */
If I didn't work for Intel, I'd wonder what the heck CET is and what the
heck it has to do with _PAGE_DIRTY_HW. I think we need a better comment
than this. How about:
Some processors can _start_ a write, but end up seeing
a read-only PTE by the time they get to getting the
Dirty bit. In this case, they will set the Dirty bit,
leaving a read-only, Dirty PTE which looks like a Shadow
Stack PTE.
However, this behavior has been improved and will *not* occur on
processors supporting Shadow Stacks. Without this guarantee, a
transition to a non-present PTE and flush the TLB would be
needed.
Interesting. Does that regard the knights landing bug or something more
general?
Will the write succeed or trigger a page-fault in this case?
[ I know it is not related to the patch, but I would appreciate if you share
your knowledge ]
Regards,
Nadav
From: Dave Hansen <dave.hansen@linux.intel.com> Date: 2018-07-10 23:24:44
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
+ /*
+ * Verify X86_PF_SHSTK is within a shadow stack VMA.
+ * It is always an error if there is a shadow stack
+ * fault outside a shadow stack VMA.
+ */
+ if (error_code & X86_PF_SHSTK) {
+ if (!(vma->vm_flags & VM_SHSTK))
+ return 1;
+ return 0;
+ }
It turns out that a X86_PF_SHSTK just means that the processor faulted
while doing access to something it thinks should be a shadow-stack
virtual address.
But, we *can* have faults on shadow stack accesses for non-shadow-stack
reasons.
I think you need to remove the 'return 0' and let it fall through to the
other access checks that we might be failing. If it's a shadow stack
access, it has to be a shadow stack VMA. But, a shadow-stack access
fault to a shadow stack VMA isn't _necessarily_ OK.
From: Dave Hansen <dave.hansen@linux.intel.com> Date: 2018-07-10 23:37:47
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
There are three possible shadow stack PTE settings:
Normal SHSTK PTE: (R/O + DIRTY_HW)
SHSTK PTE COW'ed: (R/O + DIRTY_HW)
SHSTK PTE shared as R/O data: (R/O + DIRTY_SW)
Update can_follow_write_pte/pmd for the shadow stack.
First of all, thanks for the excellent patch headers. It's nice to have
that reference every time even though it's repeated.
Can we just pass the VMA in here? This use is OK-ish, but I generally
detest true/false function arguments because you can't tell what they
are when they show up without a named variable.
But... Why does this even matter? Your own example showed that all
shadowstack PTEs have either DIRTY_HW or DIRTY_SW set, and pte_dirty()
checks both.
That makes this check seem a bit superfluous.
From: Dave Hansen <dave.hansen@linux.intel.com> Date: 2018-07-10 23:49:35
+/*
+ * WRUSS is a kernel instrcution and but writes to user
+ * shadow stack memory. When a fault occurs, both
+ * X86_PF_USER and X86_PF_SHSTK are set.
+ */
+static int is_wruss(struct pt_regs *regs, unsigned long error_code)
+{
+ return (((error_code & (X86_PF_USER | X86_PF_SHSTK)) ==
+ (X86_PF_USER | X86_PF_SHSTK)) && !user_mode(regs));
+}
I thought X86_PF_USER was set based on the mode in which the fault
occurred. Does this mean that the architecture of this bit is different
now?
That seems like something we need to call out if so. It also means we
need to update the SDM because some of the text is wrong.
quoted hunk
static void
show_fault_oops(struct pt_regs *regs, unsigned long error_code,
unsigned long address)
@@ -848,7 +859,7 @@ __bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code, struct task_struct *tsk = current; /* User mode accesses just cause a SIGSEGV */- if (error_code & X86_PF_USER) {+ if ((error_code & X86_PF_USER) && !is_wruss(regs, error_code)) { /* * It's possible to have interrupts off here: */
This needs commenting about why is_wruss() is special.
From: Dave Hansen <dave.hansen@linux.intel.com> Date: 2018-07-10 23:52:43
On 07/10/2018 04:23 PM, Nadav Amit wrote:
at 6:44 PM, Dave Hansen [off-list ref] wrote:
quoted
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
quoted
+ /*
+ * On platforms before CET, other threads could race to
+ * create a RO and _PAGE_DIRTY_HW PMD again. However,
+ * on CET platforms, this is safe without a TLB flush.
+ */
If I didn't work for Intel, I'd wonder what the heck CET is and what the
heck it has to do with _PAGE_DIRTY_HW. I think we need a better comment
than this. How about:
Some processors can _start_ a write, but end up seeing
a read-only PTE by the time they get to getting the
Dirty bit. In this case, they will set the Dirty bit,
leaving a read-only, Dirty PTE which looks like a Shadow
Stack PTE.
However, this behavior has been improved and will *not* occur on
processors supporting Shadow Stacks. Without this guarantee, a
transition to a non-present PTE and flush the TLB would be
needed.
Interesting. Does that regard the knights landing bug or something more
general?
It's more general.
Will the write succeed or trigger a page-fault in this case?
From: Dave Hansen <dave.hansen@linux.intel.com> Date: 2018-07-10 23:57:45
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
The indirect branch tracking legacy bitmap takes a large address
space. This causes may_expand_vm() failure on the address limit
check. For a IBT-enabled task, add the bitmap size to the
address limit.
This appears to require that we set up
current->thread.cet.ibt_bitmap_size _before_ calling may_expand_vm().
What keeps the ibt_mmap() itself from hitting the address limit?
We're going to have to start consolidating these at some point. We have
at least three of them now, maybe more.
+int cet_setup_ibt_bitmap(void)
+{
+ u64 r;
+ unsigned long bitmap;
+ unsigned long size;
+
+ if (!cpu_feature_enabled(X86_FEATURE_IBT))
+ return -EOPNOTSUPP;
+
+ size = TASK_SIZE_MAX / PAGE_SIZE / BITS_PER_BYTE;
Just a note: this table is going to be gigantic on 5-level paging
systems, and userspace won't, by default use any of that extra address
space. I think it ends up being a 512GB allocation in a 128TB address
space.
Is that a problem?
On 5-level paging systems, maybe we should just stick it up in the high
part of the address space.
@@ -222,7 +223,8 @@ int arch_setup_features(void *ehdr_p, void *phdr_p,structelf64_hdr*ehdr64=ehdr_p;-if(!cpu_feature_enabled(X86_FEATURE_SHSTK))+if(!cpu_feature_enabled(X86_FEATURE_SHSTK)&&+!cpu_feature_enabled(X86_FEATURE_IBT))return0;if(ehdr64->e_ident[EI_CLASS]==ELFCLASS64){
@@ -250,6 +252,9 @@ int arch_setup_features(void *ehdr_p, void *phdr_p,current->thread.cet.shstk_enabled=0;current->thread.cet.shstk_base=0;current->thread.cet.shstk_size=0;+current->thread.cet.ibt_enabled=0;+current->thread.cet.ibt_bitmap_addr=0;+current->thread.cet.ibt_bitmap_size=0;if(cpu_feature_enabled(X86_FEATURE_SHSTK)){if(shstk){err=cet_setup_shstk();
@@ -257,6 +262,15 @@ int arch_setup_features(void *ehdr_p, void *phdr_p,gotoout;}}++if(cpu_feature_enabled(X86_FEATURE_IBT)){+if(ibt){+err=cet_setup_ibt();+if(err<0)+gotoout;+}+}
You introduced 'ibt' before it was used. Please wait to introduce it
until you actually use it to make it easier to review.
Also, what's wrong with:
if (cpu_feature_enabled(X86_FEATURE_IBT) && ibt) {
...
}
?
We normally use .rst for this kind of formatted text.
+[6] The implementation of the SHSTK
+===================================
+
+SHSTK size
+----------
+
+A task's SHSTK is allocated from memory to a fixed size that can
+support 32 KB nested function calls; that is 256 KB for a 64-bit
+application and 128 KB for a 32-bit application. The system admin
+can change the default size.
From: Peter Zijlstra <peterz@infradead.org> Date: 2018-07-11 08:34:47
On Tue, Jul 10, 2018 at 03:26:20PM -0700, Yu-cheng Yu wrote:
VM_SHSTK indicates a shadow stack memory area.
A shadow stack PTE must be read-only and dirty. For non shadow
stack, we use a spare bit of the 64-bit PTE for dirty. The PTE
changes are in the next patch.
This doesn't make any sense.. the $subject and the patch seem completely
unrelated to this Changelog.
From: Peter Zijlstra <peterz@infradead.org> Date: 2018-07-11 08:48:54
On Tue, Jul 10, 2018 at 03:44:32PM -0700, Dave Hansen wrote:
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
quoted
+ /*
+ * On platforms before CET, other threads could race to
+ * create a RO and _PAGE_DIRTY_HW PMD again. However,
+ * on CET platforms, this is safe without a TLB flush.
+ */
If I didn't work for Intel, I'd wonder what the heck CET is and what the
heck it has to do with _PAGE_DIRTY_HW. I think we need a better comment
And Changelog, the provided one is abysmal.
than this. How about:
Some processors can _start_ a write, but end up seeing
a read-only PTE by the time they get to getting the
Dirty bit. In this case, they will set the Dirty bit,
leaving a read-only, Dirty PTE which looks like a Shadow
Stack PTE.
However, this behavior has been improved and will *not* occur on
processors supporting Shadow Stacks. Without this guarantee, a
transition to a non-present PTE and flush the TLB would be
needed.
I'm still struggling. I think I get the first paragraph, but then what?
Do we want to lift this hunk of code and put it elsewhere? Maybe:
entry = pte_set_vma_features(entry, vma);
and then:
pte_t pte_set_vma_features(pte_t entry, struct vm_area_struct)
{
/*
* Shadow stack PTEs are always dirty and always
* writable. They have a different encoding for
* this than normal PTEs, though.
*/
if (is_shstk_mapping(vma->vm_flags))
entry = pte_mkdirty_shstk(entry);
else
entry = pte_mkdirty(entry);
entry = maybe_mkwrite(entry, vma);
return entry;
}
Yes, that wants a helper like that. Not sold on the name, but whatever.
Is there any way we can hide all the shadow stack magic in arch code?
From: Peter Zijlstra <peterz@infradead.org> Date: 2018-07-11 09:12:43
On Tue, Jul 10, 2018 at 04:10:08PM -0700, Dave Hansen wrote:
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
quoted
Signed-off-by: Yu-cheng Yu <redacted>
This still needs a changelog, even if you think it's simple.
quoted
--- a/mm/mprotect.c+++ b/mm/mprotect.c
@@ -446,6 +446,15 @@ static int do_mprotect_pkey(unsigned long start, size_t len,error=-ENOMEM;if(!vma)gotoout;++/*+*Donotallowchangingshadowstackmemory.+*/+if(vma->vm_flags&VM_SHSTK){+error=-EINVAL;+gotoout;+}+
I think this is a _bit_ draconian. Why shouldn't we be able to use
protection keys with a shadow stack? Or, set it to PROT_NONE?
Right, and then there's also madvise() and some of the other accessors.
Why do we need to disallow this? AFAICT the worst that can happen is
that a process wrecks itself, so what?
That's against naming convention here.
static inline bool pte_shstk(pte_t pte)
{
return pte_flags(pte) & (_PAGE_RW | _PAGE_DIRTY_HW) == _PAGE_DIRTY_HW;
}
would be more in style with the rest of this code.
From: Peter Zijlstra <peterz@infradead.org> Date: 2018-07-11 09:30:02
On Tue, Jul 10, 2018 at 03:26:28PM -0700, Yu-cheng Yu wrote:
There are three possible shadow stack PTE settings:
Normal SHSTK PTE: (R/O + DIRTY_HW)
SHSTK PTE COW'ed: (R/O + DIRTY_HW)
SHSTK PTE shared as R/O data: (R/O + DIRTY_SW)
I count _2_ distinct states there.
Update can_follow_write_pte/pmd for the shadow stack.
So the below disallows can_follow_write when shstk && _PAGE_DIRTY_SW,
but this here Changelog doesn't explain why. Doesn't even get close.
Also, the code is a right mess :/ Can't we try harder to not let this
shadow stack stuff escape arch code.
From: Peter Zijlstra <peterz@infradead.org> Date: 2018-07-11 09:45:06
On Tue, Jul 10, 2018 at 03:26:30PM -0700, Yu-cheng Yu wrote:
WRUSS is a new kernel-mode instruction but writes directly
to user shadow stack memory. This is used to construct
a return address on the shadow stack for the signal
handler.
This instruction can fault if the user shadow stack is
invalid shadow stack memory. In that case, the kernel does
fixup.
+static inline int write_user_shstk_64(unsigned long addr, unsigned long val)
+{
+ int err = 0;
+
+ asm volatile("1: wrussq %[val], (%[addr])\n"
+ "xor %[err], %[err]\n"
this XOR is superfluous, you already cleared @err above.
+To build a CET-enabled kernel, Binutils v2.30 and GCC v8.1 or later
+are required. To build a CET-enabled application, GLIBC v2.29 or
+later is also requried.
Have you given up on getting the required changes into glibc 2.28?
Thanks,
Florian
Ok, could we first please make this part of the regset code more readable and
start the series with a standalone clean-up patch that changes these initializers
to something more readable:
[REGSET_CET64] = {
.core_note_type = NT_X86_CET,
.n = sizeof(struct cet_user_state) / sizeof(u64),
.size = sizeof(u64),
.align = sizeof(u64),
.active = cetregs_active,
.get = cetregs_get,
.set = cetregs_set
},
? (I'm demonstrating the cleanup based on REGSET_CET64, but this should be done on
every other entry first.)
@@ -401,6 +401,7 @@ typedef struct elf64_shdr {#define NT_386_TLS 0x200 /* i386 TLS slots (struct user_desc) */#define NT_386_IOPERM 0x201 /* x86 io permission bitmap (1=deny) */#define NT_X86_XSTATE 0x202 /* x86 extended state using xsave */+#define NT_X86_CET 0x203 /* x86 cet state */
Acronyms in comments should be in capital letters.
Also, I think I asked this before: why does "Control Flow Enforcement" abbreviate
to "CET" (which is a well-known acronym for "Central European Time"), not to CFE?
Thanks,
Ingo
+ /*
+ * PT_NOTE segment is small. Read at most
+ * PAGE_SIZE.
+ */
+ if (note_size > PAGE_SIZE)
+ note_size = PAGE_SIZE;
That's not really true. There are some huge PT_NOTE segments out there.
Why can't you check the notes after the executable has been mapped?
Thanks,
Florian
arch_prctl(ARCH_CET_DISABLE, unsigned long features)
Disable SHSTK and/or IBT specified in 'features'. Return -EPERM
if CET is locked out.
arch_prctl(ARCH_CET_LOCK)
Lock out CET feature.
Isn't it a “lock in” rather than a “lock out”?
Thanks,
Florian
On Wed, Jul 11, 2018 at 2:57 AM, Florian Weimer [off-list ref] wrote:
On 07/11/2018 12:26 AM, Yu-cheng Yu wrote:
quoted
+To build a CET-enabled kernel, Binutils v2.30 and GCC v8.1 or later
+are required. To build a CET-enabled application, GLIBC v2.29 or
+later is also requried.
Have you given up on getting the required changes into glibc 2.28?
This is a typo. We are still targeting for 2.28. All pieces are there.
--
H.J.
On Wed, Jul 11, 2018 at 2:57 AM, Florian Weimer [off-list ref]
wrote:
quoted
On 07/11/2018 12:26 AM, Yu-cheng Yu wrote:
quoted
+To build a CET-enabled kernel, Binutils v2.30 and GCC v8.1 or
later
+are required. To build a CET-enabled application, GLIBC v2.29
or
+later is also requried.
Have you given up on getting the required changes into glibc 2.28?
This is a typo. We are still targeting for 2.28. All pieces are
there.
On Wed, 2018-07-11 at 11:44 +0200, Peter Zijlstra wrote:
On Tue, Jul 10, 2018 at 03:26:30PM -0700, Yu-cheng Yu wrote:
quoted
WRUSS is a new kernel-mode instruction but writes directly
to user shadow stack memory. This is used to construct
a return address on the shadow stack for the signal
handler.
This instruction can fault if the user shadow stack is
invalid shadow stack memory. In that case, the kernel does
fixup.
+static inline int write_user_shstk_64(unsigned long addr, unsigned
long val)
+{
+ int err = 0;
+
+ asm volatile("1: wrussq %[val], (%[addr])\n"
+ "xor %[err], %[err]\n"
this XOR is superfluous, you already cleared @err above.
We normally use .rst for this kind of formatted text.
I will change this to a .rst file.
quoted
+[6] The implementation of the SHSTK
+===================================
+
+SHSTK size
+----------
+
+A task's SHSTK is allocated from memory to a fixed size that can
+support 32 KB nested function calls; that is 256 KB for a 64-bit
+application and 128 KB for a 32-bit application. The system admin
+can change the default size.
How does admin change that? We already have ulimit for stack size,
should those be somehow tied together?
$ ulimit -a
...
stack size (kbytes, -s) 8192
Ok, could we first please make this part of the regset code more
readable and
start the series with a standalone clean-up patch that changes these
initializers
to something more readable:
[REGSET_CET64] = {
.core_note_type = NT_X86_CET,
.n = sizeof(struct cet_user_state) /
sizeof(u64),
.size = sizeof(u64),
.align = sizeof(u64),
.active = cetregs_active,
.get = cetregs_get,
.set = cetregs_set
},
? (I'm demonstrating the cleanup based on REGSET_CET64, but this
should be done on
every other entry first.)
#define NT_386_TLS 0x200 /* i386 TLS slots
(struct user_desc) */
#define NT_386_IOPERM 0x201 /* x86 io
permission bitmap (1=deny) */
#define NT_X86_XSTATE 0x202 /* x86 extended
state using xsave */
+#define NT_X86_CET 0x203 /* x86 cet state */
Acronyms in comments should be in capital letters.
Also, I think I asked this before: why does "Control Flow
Enforcement" abbreviate
to "CET" (which is a well-known acronym for "Central European Time"),
not to CFE?
I don't know if I can change that, will find out.
Thanks,
Yu-cheng
On Wed, 2018-07-11 at 11:12 +0200, Peter Zijlstra wrote:
On Tue, Jul 10, 2018 at 04:10:08PM -0700, Dave Hansen wrote:
quoted
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
quoted
Signed-off-by: Yu-cheng Yu <redacted>
This still needs a changelog, even if you think it's simple.
quoted
--- a/mm/mprotect.c+++ b/mm/mprotect.c
@@ -446,6 +446,15 @@ static int do_mprotect_pkey(unsigned long
start, size_t len,
error = -ENOMEM;
if (!vma)
goto out;
+
+ /*
+ * Do not allow changing shadow stack memory.
+ */
+ if (vma->vm_flags & VM_SHSTK) {
+ error = -EINVAL;
+ goto out;
+ }
+
I think this is a _bit_ draconian. Why shouldn't we be able to use
protection keys with a shadow stack? Or, set it to PROT_NONE?
Right, and then there's also madvise() and some of the other
accessors.
Why do we need to disallow this? AFAICT the worst that can happen is
that a process wrecks itself, so what?
@@ -4108,7 +4108,13 @@ static int __handle_mm_fault(struct
vm_area_struct *vma, unsigned long address,
if (pmd_protnone(orig_pmd) &&
vma_is_accessible(vma))
return do_huge_pmd_numa_page(&vmf,
orig_pmd);
- if (dirty && !pmd_write(orig_pmd)) {
+ /*
+ * Shadow stack trans huge PMDs are copy-
on-access,
+ * so wp_huge_pmd() on them no mater if we
have a
+ * write fault or not.
+ */
+ if (is_shstk_mapping(vma->vm_flags) ||
+ (dirty && !pmd_write(orig_pmd))) {
ret = wp_huge_pmd(&vmf, orig_pmd);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
Can't we do this (and the do_wp_page thing) by setting
FAULT_FLAG_WRITE
in the arch fault handler on shadow stack faults?
This can work. I don't know if that will create other issues.
Let me think about that.
Yu-cheng
On Wed, 2018-07-11 at 10:34 +0200, Peter Zijlstra wrote:
On Tue, Jul 10, 2018 at 03:26:20PM -0700, Yu-cheng Yu wrote:
quoted
VM_SHSTK indicates a shadow stack memory area.
A shadow stack PTE must be read-only and dirty. For non shadow
stack, we use a spare bit of the 64-bit PTE for dirty. The PTE
changes are in the next patch.
This doesn't make any sense.. the $subject and the patch seem
completely
unrelated to this Changelog.
I was trying to say why this is only defined for 64-bit. I will fix
it.
Yu-cheng
From: Dave Hansen <dave.hansen@linux.intel.com> Date: 2018-07-11 16:22:32
On 07/11/2018 09:07 AM, Yu-cheng Yu wrote:
quoted
Why do we need to disallow this? AFAICT the worst that can happen is
that a process wrecks itself, so what?
Agree. I will remove the patch.
No so quick. :)
We still need to find out a way to handle things that ask for an
mprotect() which is incompatible with shadow stacks. PROT_READ without
PROT_WRITE comes to mind. We also need to be careful that we don't
copy-on-write/copy-on-access pages which fault on PROT_NONE. I *think*
it'll get done correctly but we have to be sure.
BTW, where are all the selftests for this code? We're slowly building
up a list of pathological things that need to get tested.
I don't think this can or should get merged before we have selftests.
On Tue, 2018-07-10 at 16:57 -0700, Dave Hansen wrote:
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
quoted
The indirect branch tracking legacy bitmap takes a large address
space. This causes may_expand_vm() failure on the address limit
check. For a IBT-enabled task, add the bitmap size to the
address limit.
This appears to require that we set up
current->thread.cet.ibt_bitmap_size _before_ calling may_expand_vm().
What keeps the ibt_mmap() itself from hitting the address limit?
Yes, that is overlooked. I will fix it.
Thanks,
Yu-cheng
On Tue, 2018-07-10 at 16:37 -0700, Dave Hansen wrote:
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
quoted
There are three possible shadow stack PTE settings:
Normal SHSTK PTE: (R/O + DIRTY_HW)
SHSTK PTE COW'ed: (R/O + DIRTY_HW)
SHSTK PTE shared as R/O data: (R/O + DIRTY_SW)
Update can_follow_write_pte/pmd for the shadow stack.
First of all, thanks for the excellent patch headers. It's nice to
have
that reference every time even though it's repeated.
Can we just pass the VMA in here? This use is OK-ish, but I
generally
detest true/false function arguments because you can't tell what they
are when they show up without a named variable.
But... Why does this even matter? Your own example showed that all
shadowstack PTEs have either DIRTY_HW or DIRTY_SW set, and
pte_dirty()
checks both.
That makes this check seem a bit superfluous.
My understanding is that we don't want to follow write pte if the page
is shared as read-only. For a SHSTK page, that is (R/O + DIRTY_SW),
which means the SHSTK page has not been COW'ed. Is that right?
Thanks,
Yu-cheng
On Tue, 2018-07-10 at 15:52 -0700, Dave Hansen wrote:
On 07/10/2018 03:26 PM, Yu-cheng Yu wrote:
quoted
+++ b/arch/x86/include/asm/traps.h
@@ -157,6 +157,7 @@ enum {
* bit 3 == 1: use of reserved
bit detected
* bit 4 == 1: fault was an
instruction fetch
* bit 5 == 1: protection keys
block access
+ * bit 6 == 1: shadow stack
access fault
*/
Could we document this bit better?
Is this a fault where the *processor* thought it should be a shadow
stack fault? Or is it also set on faults to valid shadow stack PTEs
that just happen to fault for other reasons, say protection keys?
Thanks Vedvyas for explaining this to me.
I will add this to comments:
This flag is 1 if (1) CR4.CET = 1; and (2) the access causing the page-
fault exception was a shadow-stack data access.
So this bit does not report the reason for the fault. It reports the
type of access; i.e. it was a shadow-stack-load or a shadow-stack-store
that took the page fault. The fault could have been caused by any
variety of reasons including protection keys.