If we ever add kernel support, "no_cet_shstk" will mean "no cet shstk
for userspace"?

+	no_cet_ibt	[X86-64] Disable Indirect Branch Tracking for user-mode
+			applications
+
 	nosmap		[X86,PPC]
 			Disable SMAP (Supervisor Mode Access Prevention)
 			even if it is supported by processor.

BTW, this documentation is misplaced.  It needs to go to the spot where
you introduce the code for these options.

							      ^ set up

+protect both applications and the kernel.  In the first phase, only
+user-mode protection is implemented in the 64-bit kernel; 32-bit
+applications are supported in compatibility mode.

Please just say what *is* at the time of the writing.  We don't need to
talk about "phases".

Also, you haven't mentioned that this is a *hardware* feature and that
it's only on Intel CPUs at the moment.  That's kinda essential.  If I've
got an AMD CPU, I can just stop reading. :)

The hardware supports shadow stacks for both userspace and the kernel in
both 32 and 64-bit modes.  32-bit kernel support is not implemented.
Both 32-bit and 64-bit user applications can run on 64-bit kernels.

This is also missing the same key points about enabling as the Kconfig
text: apps don't get this for free and must be specifically enabled.

+CET introduces Shadow Stack (SHSTK) and Indirect Branch Tracking
+(IBT).  SHSTK is a secondary stack allocated from memory and cannot
+be directly modified by applications.  When executing a CALL, the
+processor pushes a copy of the return address to SHSTK. 

... and to the normal stack

 Upon
+function return, the processor pops the SHSTK copy and compares it
+to the one from the program stack.  If the two copies differ, the
+processor raises a control-protection fault.  IBT verifies indirect
+CALL/JMP targets are intended as marked by the compiler with 'ENDBR'
+opcodes (see CET instructions below).
+
+There are two kernel configuration options:
+
+    X86_INTEL_SHADOW_STACK_USER, and
+    X86_INTEL_BRANCH_TRACKING_USER.
+
+To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or later
+are required.

Why are these needed to build a CET-enabled kernel?

 To build a CET-enabled application, GLIBC v2.28 or
+later is also required.
+
+There are two command-line options for disabling CET features::
+
+    no_cet_shstk - disables SHSTK, and
+    no_cet_ibt   - disables IBT.
+
+At run time, /proc/cpuinfo shows the availability of SHSTK and IBT.

Availability of what?

If I set X86_INTEL_SHADOW_STACK_USER=n, I'll still see the cpuinfo
flags, but I won't have runtime support.

Probably best to say that cpuinfo tells you about processor support only.

+[2] CET assembly instructions
+=============================

Why do we need this in the kernel?  What is specific to Linux or the
kernel?  Why wouldn't I just go read the SDM if I want to know how the
instructions work?

+[3] Application Enabling
+========================
+
+An application's CET capability is marked in its ELF header and can
+be verified from the following command output, in the
+NT_GNU_PROPERTY_TYPE_0 field:
+
+    readelf -n <application>
+
+If an application supports CET and is statically linked, it will run
+with CET protection.  If the application needs any shared libraries,
+the loader checks all dependencies and enables CET only when all
+requirements are met.

What about shared libraries loaded after the program starts?

+[4] Legacy Libraries
+====================
+
+GLIBC provides a few tunables for backward compatibility.
+
+GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT
+    Turn off SHSTK/IBT for the current shell.
+
+GLIBC_TUNABLES=glibc.tune.x86_shstk=<on, permissive>
+    This controls how dlopen() handles SHSTK legacy libraries::
+
+        on         - continue with SHSTK enabled;
+        permissive - continue with SHSTK off.

This seems like manpage fodder more than kernel documentation to me.

+[5] CET system calls
+====================
+
+The following arch_prctl() system calls are added for CET:

FWIW, I wouldn't call each of these a "system call".

"Several arch_prctl()'s have been added for CET:"

+arch_prctl(ARCH_X86_CET_STATUS, unsigned long *addr)
+    Return CET feature status.
+
+    The parameter 'addr' is a pointer to a user buffer.
+    On returning to the caller, the kernel fills the following
+    information::
+
+        *addr       = SHSTK/IBT status
+        *(addr + 1) = SHSTK base address
+        *(addr + 2) = SHSTK size
+
+arch_prctl(ARCH_X86_CET_DISABLE, unsigned long features)
+    Disable SHSTK and/or IBT specified in 'features'.  Return -EPERM
+    if CET is locked.
+
+arch_prctl(ARCH_X86_CET_LOCK)
+    Lock in CET feature.

Shouldn't this say what "locking" means?

+arch_prctl(ARCH_X86_CET_ALLOC_SHSTK, unsigned long *addr)
+    Allocate a new SHSTK and put a restore token at top.
+
+    The parameter 'addr' is a pointer to a user buffer and indicates
+    the desired SHSTK size to allocate.  On returning to the caller,
+    the kernel fills '*addr' with the base address of the new SHSTK.

+arch_prctl(ARCH_X86_CET_MARK_LEGACY_CODE, unsigned long *addr)
+    Mark an address range as IBT legacy code.
+
+    The parameter 'addr' is a pointer to a user buffer that has the
+    following information::
+
+        *addr       = starting linear address of the legacy code
+        *(addr + 1) = size of the legacy code
+        *(addr + 2) = set (1); clear (0)
+
+Note:
+  There is no CET-enabling arch_prctl function.  By design, CET is
+  enabled automatically if the binary and the system can support it.

This is kinda interesting.  It means that a JIT couldn't choose to
protect the code it generates and have different rules from itself?

+  The parameters passed are always unsigned 64-bit.  When an IA32
+  application passing pointers, it should only use the lower 32 bits.

Won't a 32-bit app calling prctl() use the 32-bit ABI?  How would it
even know it's running on a 64-bit kernel?

+[6] The implementation of the SHSTK
+===================================
+
+SHSTK size
+----------
+
+A task's SHSTK is allocated from memory to a fixed size of
+RLIMIT_STACK.

I can't really parse that sentence.  Is this saying that shadow stacks
are limited by and share space with normal stacks via RLIMIT_STACK?

 A compat-mode thread's SHSTK size is 1/4 of
+RLIMIT_STACK.  The smaller 32-bit thread SHSTK allows more threads to
+share a 32-bit address space.

I thought the size was passed in from userspace?  Where does this sizing
take place?  Is this a convention or is it being enforced?

+Signal
+------
+
+The main program and its signal handlers use the same SHSTK.  Because
+the SHSTK stores only return addresses, a large SHSTK will cover the
+condition that both the program stack and the sigaltstack run out.

						 ^ typo?

I'm not sure what this is trying to say.

+The kernel creates a restore token at the SHSTK restoring address and
+verifies that token when restoring from the signal handler.

I think there's a sentence or two of background missing here.  I'm
really lost as to what this is trying to tell me.

+IBT for signal delivering and sigreturn is the same as the main
+program's setup; except for WAIT_ENDBR status, which can be read from
+MSR_IA32_U_CET.  In general, a task is in WAIT_ENDBR after an
+indirect CALL/JMP and before the next instruction starts.

I'm 100% lost.  I have no idea what this is trying to tell me or why it
is relevant to the kernel.

+Fork
+----
+
+The SHSTK's vma has VM_SHSTK flag set; its PTEs are required to be
+read-only and dirty.  When a SHSTK PTE is not present, RO, and dirty,
+a SHSTK access triggers a page fault with an additional SHSTK bit set
+in the page fault error code.
+
+When a task forks a child, its SHSTK PTEs are copied and both the
+parent's and the child's SHSTK PTEs are cleared of the dirty bit.
+Upon the next SHSTK access, the resulting SHSTK page fault is handled
+by page copy/re-use.

What's the most important thing about shadow stacks and fork()?  Does
this documentation tell that to the reader?

+When a pthread child is created, the kernel allocates a new SHSTK for
+the new thread.

Why is this here?  Are pthread children created work fork()?

+Setjmp/Longjmp
+--------------
+
+Longjmp unwinds SHSTK until it matches the program stack.

I'm missing what this has to do with the kernel.

+Ucontext
+--------
+
+In GLIBC, getcontext/setcontext is implemented in similar way as
+setjmp/longjmp.
+
+When makecontext creates a new ucontext, a new SHSTK is allocated for
+that context with ARCH_X86_CET_ALLOC_SHSTK syscall.  The kernel

Nit: ARCH_X86_CET_ALLOC_SHSTK is not a syscall.

+creates a restore token at the top of the new SHSTK and the user-mode
+code switches to the new SHSTK with the RSTORSSP instruction.

This seems like a howto for doing user-level threading.  It seems like
it could be replaced by a single sentence in the
ARCH_X86_CET_ALLOC_SHSTK documentation explaining that new shadow stacks
are generally (always??) allocated along with new stacks.  Since new
clone() threads need a new stack, they also need a new shadow stack.
User-level threads that need a new stack are also expected to allocate a
new shadow stack.

Right?

+[7] The management of read-only & dirty PTEs for SHSTK
+======================================================
+
+A RO and dirty PTE exists in the following cases:
+
+(a) A page is modified and then shared with a fork()'ed child;
+(b) A R/O page that has been COW'ed;
+(c) A SHSTK page.
+
+The processor only checks the dirty bit for (c).  To prevent the use
+of non-SHSTK memory as SHSTK, we use a spare bit of the 64-bit PTE as
+DIRTY_SW for (a) and (b) above.  This results to the following PTE
+settings::
+
+    Modified PTE:             (R/W + DIRTY_HW)
+    Modified and shared PTE:  (R/O + DIRTY_SW)
+    R/O PTE, COW'ed:          (R/O + DIRTY_SW)
+    SHSTK PTE:                (R/O + DIRTY_HW)
+    SHSTK PTE, COW'ed:        (R/O + DIRTY_HW)
+    SHSTK PTE, shared:        (R/O + DIRTY_SW)
+
+Note that DIRTY_SW is only used in R/O PTEs but not R/W PTEs.

I really don't think this belongs in the documentation, especially since
it's duplicated almost verbatim in code comments.

+[8] The implementation of IBT legacy bitmap
+===========================================
+
+When IBT is active, a non-IBT-capable legacy library can be executed
+if its address ranges are specified in the legacy code bitmap.  The
+bitmap covers the whole user-space address, which is TASK_SIZE_MAX
+for 64-bit and TASK_SIZE for IA32, and its each bit indicates a 4-KB
+legacy code page.  It is read-only from an application, and setup by

								^ set up

+the kernel as a special mapping when the first time the application
+calls arch_prctl(ARCH_X86_CET_MARK_LEGACY_CODE).  The application
+manages the bitmap through the arch_prctl.