This series implements C syscall entry/exit for PPC32. It reuses
the work already done for PPC64.
This series is based on Nick's v6 series "powerpc: interrupt wrappers".
Patch 1 is a bug fix submitted separately but this series depends on it.
Patches 2-4 are an extract from the series "powerpc/32: Reduce head
complexity and re-activate MMU earlier". The changes here are limited
to system calls. That series will be respined to only contain exception
related changes and the syscall changes will remain in this series.
Patches 5-16 are preparatory changes.
Patch 17 is THE patch that changes to C syscall entry/exit
Patches 18-23 are optimisations.
In terms on performance we have the following number of cycles on an
8xx running null_syscall benchmark:
- mainline: 296 cycles
- after patch 4: 283 cycles
- after patch 16: 304 cycles
- after patch 17: 348 cycles
- at the end of the series: 320 cycles
So in summary, we have a degradation of performance of 8% on null_syscall.
I think it is not a big degradation, it is worth it.
v4 is the first mature version.
Christophe Leroy (23):
powerpc/32s: Add missing call to kuep_lock on syscall entry
powerpc/32: Always enable data translation on syscall entry
powerpc/32: On syscall entry, enable instruction translation at the
same time as data
powerpc/32: Reorder instructions to avoid using CTR in syscall entry
powerpc/64s: Make kuap_check_amr() and kuap_get_and_check_amr()
generic
powerpc/32s: Create C version of kuap_user/kernel_restore() and
friends
powerpc/8xx: Create C version of kuap_user/kernel_restore() and
friends
powerpc/irq: Add helper to set regs->softe
powerpc/irq: Rework helpers that manipulate MSR[EE/RI]
powerpc/irq: Add stub irq_soft_mask_return() for PPC32
powerpc/syscall: Rename syscall_64.c into syscall.c
powerpc/syscall: Make syscall.c buildable on PPC32
powerpc/syscall: Use is_compat_task()
powerpc/syscall: Save r3 in regs->orig_r3
powerpc/syscall: Change condition to check MSR_RI
powerpc/32: Always save non volatile GPRs at syscall entry
powerpc/syscall: implement system call entry/exit logic in C for PPC32
powerpc/32: Remove verification of MSR_PR on syscall in the ASM entry
powerpc/syscall: Avoid stack frame in likely part of
system_call_exception()
powerpc/syscall: Do not check unsupported scv vector on PPC32
powerpc/syscall: Remove FULL_REGS verification in
system_call_exception
powerpc/syscall: Optimise checks in beginning of
system_call_exception()
powerpc/syscall: Avoid storing 'current' in another pointer
arch/powerpc/include/asm/book3s/32/kup.h | 33 ++
arch/powerpc/include/asm/book3s/64/kup.h | 24 +-
arch/powerpc/include/asm/hw_irq.h | 91 ++++--
arch/powerpc/include/asm/kup.h | 9 +-
arch/powerpc/include/asm/nohash/32/kup-8xx.h | 27 ++
arch/powerpc/include/asm/reg.h | 1 +
arch/powerpc/kernel/Makefile | 4 +-
arch/powerpc/kernel/entry_32.S | 305 ++----------------
arch/powerpc/kernel/entry_64.S | 1 -
arch/powerpc/kernel/head_32.h | 76 +----
arch/powerpc/kernel/head_booke.h | 27 +-
.../kernel/{syscall_64.c => syscall.c} | 57 ++--
arch/powerpc/kernel/syscalls/syscall.tbl | 20 +-
13 files changed, 225 insertions(+), 450 deletions(-)
rename arch/powerpc/kernel/{syscall_64.c => syscall.c} (90%)
--
2.25.0
In preparation of porting powerpc32 to C syscall entry/exit,
rename kuap_check_amr() and kuap_get_and_check_amr() as kuap_check()
and kuap_get_and_check(), and move in the generic asm/kup.h the stub
for when CONFIG_PPC_KUAP is not selected.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/book3s/64/kup.h | 24 ++----------------------
arch/powerpc/include/asm/kup.h | 9 ++++++++-
arch/powerpc/kernel/syscall_64.c | 12 ++++++------
3 files changed, 16 insertions(+), 29 deletions(-)
@@ -65,7 +65,7 @@ notrace long system_call_exception(long r3, long r4, long r5,isync();}else#endif-kuap_check_amr();+kuap_check();account_cpu_user_entry();
@@ -181,7 +181,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,CT_WARN_ON(ct_state()==CONTEXT_USER);-kuap_check_amr();+kuap_check();regs->result=r3;
In preparation of porting PPC32 to C syscall entry/exit,
create C version of kuap_user_restore() and kuap_kernel_restore()
and kuap_check() and kuap_get_and_check() on book3s/32.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/book3s/32/kup.h | 33 ++++++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/arch/powerpc/kernel/syscall_64.c b/arch/powerpc/kernel/syscall.csimilarity index 100%rename from arch/powerpc/kernel/syscall_64.crename to arch/powerpc/kernel/syscall.c
--
2.25.0
In preparation of porting PPC32 to C syscall entry/exit,
create C version of kuap_user_restore() and kuap_kernel_restore()
and kuap_check() and kuap_get_and_check() on 8xx
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/nohash/32/kup-8xx.h | 27 ++++++++++++++++++++
1 file changed, 27 insertions(+)
By saving the pointer pointing to thread_info.flags, gcc copies r2
in a non-volatile register.
We know 'current' doesn't change, so avoid that intermediaite pointer.
Reduces null_syscall benchmark by 2 cycles (322 => 320 cycles)
On PPC64, gcc seems to know that 'current' is not changing, and it keeps
it in a non volatile register to avoid multiple read of 'current' in paca.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/syscall.c | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
@@ -186,7 +186,6 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,structpt_regs*regs,longscv){-unsignedlong*ti_flagsp=¤t_thread_info()->flags;unsignedlongti_flags;unsignedlongret=0;
@@ -202,7 +201,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,/* Check whether the syscall is issued inside a restartable sequence */rseq_syscall(regs);-ti_flags=*ti_flagsp;+ti_flags=current_thread_info()->flags;if(unlikely(r3>=(unsignedlong)-MAX_ERRNO)&&!scv){if(likely(!(ti_flags&(_TIF_NOERROR|_TIF_RESTOREALL)))){
@@ -216,7 +215,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,ret=_TIF_RESTOREALL;elseregs->gpr[3]=r3;-clear_bits(_TIF_PERSYSCALL_MASK,ti_flagsp);+clear_bits(_TIF_PERSYSCALL_MASK,¤t_thread_info()->flags);}else{regs->gpr[3]=r3;}
@@ -228,7 +227,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,again:local_irq_disable();-ti_flags=READ_ONCE(*ti_flagsp);+ti_flags=READ_ONCE(current_thread_info()->flags);while(unlikely(ti_flags&(_TIF_USER_WORK_MASK&~_TIF_RESTORE_TM))){local_irq_enable();if(ti_flags&_TIF_NEED_RESCHED){
@@ -244,7 +243,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,do_notify_resume(regs,ti_flags);}local_irq_disable();-ti_flags=READ_ONCE(*ti_flagsp);+ti_flags=READ_ONCE(current_thread_info()->flags);}if(IS_ENABLED(CONFIG_PPC_BOOK3S)&&IS_ENABLED(CONFIG_PPC_FPU)){
@@ -28,6 +28,7 @@ notrace long system_call_exception(long r3, long r4, long r5,unsignedlongr0,structpt_regs*regs){syscall_fnf;+unsignedlongexpected_msr;regs->orig_gpr3=r3;
@@ -39,10 +40,13 @@ notrace long system_call_exception(long r3, long r4, long r5,trace_hardirqs_off();/* finish reconciling */+expected_msr=MSR_PR;if(!IS_ENABLED(CONFIG_BOOKE)&&!IS_ENABLED(CONFIG_40x))-BUG_ON(!(regs->msr&MSR_RI));-BUG_ON(!(regs->msr&MSR_PR));-BUG_ON(arch_irq_disabled_regs(regs));+expected_msr|=MSR_RI;+if(IS_ENABLED(CONFIG_PPC32))+expected_msr|=MSR_EE;+BUG_ON((regs->msr&expected_msr)^expected_msr);+BUG_ON(IS_ENABLED(CONFIG_PPC64)&&arch_irq_disabled_regs(regs));#ifdef CONFIG_PPC_PKEYif(mmu_has_feature(MMU_FTR_PKEY)){
Only PPC64 has scv. No need to check the 0x7ff0 trap on PPC32.
And ignore the scv parameter in syscall_exit_prepare (Save 14 cycles
346 => 332 cycles)
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/entry_32.S | 1 -
arch/powerpc/kernel/syscall.c | 7 +++++--
2 files changed, 5 insertions(+), 3 deletions(-)
@@ -86,7 +86,7 @@ notrace long system_call_exception(long r3, long r4, long r5,local_irq_enable();if(unlikely(current_thread_info()->flags&_TIF_SYSCALL_DOTRACE)){-if(unlikely(regs->trap==0x7ff0)){+if(IS_ENABLED(CONFIG_PPC64)&&unlikely(regs->trap==0x7ff0)){/* Unsupported scv vector */_exception(SIGILL,regs,ILL_ILLOPC,regs->nip);returnregs->gpr[3];
@@ -109,7 +109,7 @@ notrace long system_call_exception(long r3, long r4, long r5,r8=regs->gpr[8];}elseif(unlikely(r0>=NR_syscalls)){-if(unlikely(regs->trap==0x7ff0)){+if(IS_ENABLED(CONFIG_PPC64)&&unlikely(regs->trap==0x7ff0)){/* Unsupported scv vector */_exception(SIGILL,regs,ILL_ILLOPC,regs->nip);returnregs->gpr[3];
@@ -187,6 +187,9 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,unsignedlongti_flags;unsignedlongret=0;+if(IS_ENABLED(CONFIG_PPC32))+scv=0;+CT_WARN_ON(ct_state()==CONTEXT_USER);kuap_check();
@@ -115,6 +115,9 @@ notrace long system_call_exception(long r3, long r4, long r5,returnregs->gpr[3];}return-ENOSYS;+}else{+/* Restore r3 from orig_gpr3 to free up a volatile reg */+r3=regs->orig_gpr3;}/* May be faster to do array_index_nospec? */
That's port of PPC64 syscall entry/exit logic in C to PPC32.
Performancewise on 8xx:
Before : 304 cycles on null_syscall
After : 348 cycles on null_syscall
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/entry_32.S | 227 ++++---------------------------
arch/powerpc/kernel/head_32.h | 16 ---
arch/powerpc/kernel/head_booke.h | 15 --
3 files changed, 29 insertions(+), 229 deletions(-)
@@ -162,9 +162,6 @@internaldebugmodebittodothis.*/lwzr12,THREAD_DBCR0(r12)andis.r12,r12,DBCR0_IDM@h-#endif-ACCOUNT_CPU_USER_ENTRY(r2,r11,r12)-#if defined(CONFIG_40x)beq+3f/* From user and task is ptraced - load up global dbcr0 */lir12,-1/* clear all pending debug events */
@@ -176,21 +173,8 @@lwzr12,4(r11)addir12,r12,-1stwr12,4(r11)-#endif-3:-#ifdef CONFIG_TRACE_IRQFLAGS-/*-*IfMSRischangingweneedtokeepinterruptsdisabledatthispoint-*otherwisewemightrisktakinganinterruptbeforewetelllockdep-*theyareenabled.-*/-LOAD_REG_IMMEDIATE(r10,MSR_KERNEL)-rlwimir10,r9,0,MSR_EE-#else-LOAD_REG_IMMEDIATE(r10,MSR_KERNEL|MSR_EE)#endif-mtmsrr10btransfer_to_syscall/* jump to handler */99:bret_from_kernel_syscall.endm
@@ -138,7 +138,6 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)internaldebugmodebittodothis.*/lwzr12,THREAD_DBCR0(r10)andis.r12,r12,DBCR0_IDM@h-ACCOUNT_CPU_USER_ENTRY(r2,r11,r12)beq+3f/* From user and task is ptraced - load up global dbcr0 */lir12,-1/* clear all pending debug events */
@@ -157,20 +156,6 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)stwr12,4(r11)3:-#ifdef CONFIG_TRACE_IRQFLAGS-/*-*IfMSRischangingweneedtokeepinterruptsdisabledatthispoint-*otherwisewemightrisktakinganinterruptbeforewetelllockdep-*theyareenabled.-*/-lisr10,MSR_KERNEL@h-orir10,r10,MSR_KERNEL@l-rlwimir10,r9,0,MSR_EE-#else-lisr10,(MSR_KERNEL|MSR_EE)@h-orir10,r10,(MSR_KERNEL|MSR_EE)@l-#endif-mtmsrr10btransfer_to_syscall/* jump to handler */99:bret_from_kernel_syscall.endm
system_call_exception() checks MSR_PR and BUGs if a syscall
is issued from kernel mode.
No need to handle it anymore from the ASM entry code.
null_syscall reduction 2 cycles (348 => 346 cycles)
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/entry_32.S | 30 ------------------------------
arch/powerpc/kernel/head_32.h | 3 ---
arch/powerpc/kernel/head_booke.h | 3 ---
3 files changed, 36 deletions(-)
@@ -118,8 +118,6 @@.macroSYSCALL_ENTRYtrapnomfsprr9,SPRN_SRR1mfsprr10,SPRN_SRR0-andi.r11,r9,MSR_PR-beq-99fLOAD_REG_IMMEDIATE(r11,MSR_KERNEL)/* can take exceptions */lisr12,1f@horir12,r12,1f@l
@@ -176,7 +174,6 @@3:#endifbtransfer_to_syscall/* jump to handler */-99:bret_from_kernel_syscall.endm.macrosave_dar_dsisr_on_stackreg1,reg2,sp
@@ -106,10 +106,8 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)#endifmfsprr9,SPRN_SRR1BOOKE_CLEAR_BTB(r11)-andi.r11,r9,MSR_PRlwzr11,TASK_STACK-THREAD(r10)rlwinmr12,r12,0,4,2/* Clear SO bit in CR */-beq-99fALLOC_STACK_FRAME(r11,THREAD_SIZE-INT_FRAME_SIZE)stwr12,_CCR(r11)/* save various registers */mflrr12
@@ -157,7 +155,6 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)3:btransfer_to_syscall/* jump to handler */-99:bret_from_kernel_syscall.endm/* To handle the additional exception priority levels on 40x and Book-E
For book3s/64, FULL_REGS() is 'true' at all time, so the test voids.
For others, non volatile registers are saved inconditionally.
So the verification is pointless.
Should one fail to do it, it would anyway be caught by the
CHECK_FULL_REGS() in copy_thread() as we have removed the
special versions ppc_fork() and friends.
null_syscall benchmark reduction 4 cycles (332 => 328 cycles)
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/syscall.c | 1 -
1 file changed, 1 deletion(-)
@@ -42,7 +42,6 @@ notrace long system_call_exception(long r3, long r4, long r5,if(!IS_ENABLED(CONFIG_BOOKE)&&!IS_ENABLED(CONFIG_40x))BUG_ON(!(regs->msr&MSR_RI));BUG_ON(!(regs->msr&MSR_PR));-BUG_ON(!FULL_REGS(regs));BUG_ON(arch_irq_disabled_regs(regs));#ifdef CONFIG_PPC_PKEY
If the code can use a stack in vm area, it can also use a
stack in linear space.
Simplify code by removing old non VMAP stack code on PPC32 in syscall.
That means the data translation is now re-enabled early in
syscall entry in all cases, not only when using VMAP stacks.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/head_32.h | 23 +----------------------
arch/powerpc/kernel/head_booke.h | 2 --
2 files changed, 1 insertion(+), 24 deletions(-)
@@ -126,30 +125,16 @@lwzr1,TASK_STACK-THREAD(r12)beq-99faddir1,r1,THREAD_SIZE-INT_FRAME_SIZE-lir10,MSR_KERNEL&~(MSR_IR|MSR_RI)/* can take DTLB miss */+LOAD_REG_IMMEDIATE(r10,MSR_KERNEL&~(MSR_IR|MSR_RI))/* can take DTLB miss */mtmsrr10isynctovirt(r12,r12)stwr11,GPR1(r1)stwr11,0(r1)mrr11,r1-#else-andi.r11,r9,MSR_PR-lwzr11,TASK_STACK-THREAD(r12)-beq-99f-addir11,r11,THREAD_SIZE-INT_FRAME_SIZE-tophys(r11,r11)-stwr1,GPR1(r11)-stwr1,0(r11)-tovirt(r1,r11)/* set new kernel sp */-#endifmflrr10stwr10,_LINK(r11)-#ifdef CONFIG_VMAP_STACKmfctrr10-#else-mfsprr10,SPRN_SRR0-#endifstwr10,_NIP(r11)mfcrr10rlwinmr10,r10,0,4,2/* Clear SO bit in CR */
@@ -157,11 +142,7 @@#ifdef CONFIG_40xrlwinmr9,r9,0,14,12/* clear MSR_WE (necessary?) */#else-#ifdef CONFIG_VMAP_STACKLOAD_REG_IMMEDIATE(r10,MSR_KERNEL&~MSR_IR)/* can take exceptions */-#else-LOAD_REG_IMMEDIATE(r10,MSR_KERNEL&~(MSR_IR|MSR_DR))/* can take exceptions */-#endifmtmsrr10/* (except for mach check in rtas) */#endiflisr10,STACK_FRAME_REGS_MARKER@ha/* exception frame marker */
@@ -190,7 +171,6 @@lir12,-1/* clear all pending debug events */mtsprSPRN_DBSR,r12lisr11,global_dbcr0@ha-tophys(r11,r11)addir11,r11,global_dbcr0@llwzr12,0(r11)mtsprSPRN_DBCR0,r12
@@ -200,7 +180,6 @@#endif3:-tovirt_novmstackr2,r2/* set r2 to current */lisr11,transfer_to_syscall@horir11,r11,transfer_to_syscall@l#ifdef CONFIG_TRACE_IRQFLAGS
@@ -158,7 +157,6 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)stwr12,4(r11)3:-tovirt(r2,r2)/* set r2 to current */lisr11,transfer_to_syscall@horir11,r11,transfer_to_syscall@l#ifdef CONFIG_TRACE_IRQFLAGS
On 40x and 8xx, kernel text is pinned.
On book3s/32, kernel text is mapped by BATs.
Enable instruction translation at the same time as data translation, it
makes things simpler.
MSR_RI can also be set at the same time because srr0/srr1 are already
saved and r1 is set properly.
On booke, translation is always on, so at the end all PPC32
have translation on early.
This reduces null_syscall benchmark by 13 cycles on 8xx
(296 ==> 283 cycles).
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/head_32.h | 26 +++++++++-----------------
arch/powerpc/kernel/head_booke.h | 7 ++-----
2 files changed, 11 insertions(+), 22 deletions(-)
@@ -125,9 +125,13 @@lwzr1,TASK_STACK-THREAD(r12)beq-99faddir1,r1,THREAD_SIZE-INT_FRAME_SIZE-LOAD_REG_IMMEDIATE(r10,MSR_KERNEL&~(MSR_IR|MSR_RI))/* can take DTLB miss */-mtmsrr10-isync+LOAD_REG_IMMEDIATE(r10,MSR_KERNEL)/* can take exceptions */+mtsprSPRN_SRR1,r10+lisr10,1f@h+orir10,r10,1f@l+mtsprSPRN_SRR0,r10+rfi+1:tovirt(r12,r12)stwr11,GPR1(r1)stwr11,0(r1)
@@ -141,9 +145,6 @@stwr10,_CCR(r11)/* save registers */#ifdef CONFIG_40xrlwinmr9,r9,0,14,12/* clear MSR_WE (necessary?) */-#else-LOAD_REG_IMMEDIATE(r10,MSR_KERNEL&~MSR_IR)/* can take exceptions */-mtmsrr10/* (except for mach check in rtas) */#endiflisr10,STACK_FRAME_REGS_MARKER@ha/* exception frame marker */stwr2,GPR2(r11)
Now that we are using rfi instead of mtmsr to reactivate MMU, it is
possible to reorder instructions and avoid the need to use CTR for
stashing SRR0.
null_syscall on 8xx is reduced by 3 cycles (283 => 280 cycles).
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/head_32.h | 22 ++++++++++------------
1 file changed, 10 insertions(+), 12 deletions(-)
@@ -116,30 +116,28 @@.endm.macroSYSCALL_ENTRYtrapno-mfsprr12,SPRN_SPRG_THREADmfsprr9,SPRN_SRR1-mfsprr11,SPRN_SRR0-mtctrr11+mfsprr10,SPRN_SRR0andi.r11,r9,MSR_PR+beq-99f+LOAD_REG_IMMEDIATE(r11,MSR_KERNEL)/* can take exceptions */+lisr12,1f@h+orir12,r12,1f@l+mtsprSPRN_SRR1,r11+mtsprSPRN_SRR0,r12+mfsprr12,SPRN_SPRG_THREADmrr11,r1lwzr1,TASK_STACK-THREAD(r12)-beq-99f+tovirt(r12,r12)addir1,r1,THREAD_SIZE-INT_FRAME_SIZE-LOAD_REG_IMMEDIATE(r10,MSR_KERNEL)/* can take exceptions */-mtsprSPRN_SRR1,r10-lisr10,1f@h-orir10,r10,1f@l-mtsprSPRN_SRR0,r10rfi1:-tovirt(r12,r12)stwr11,GPR1(r1)stwr11,0(r1)mrr11,r1+stwr10,_NIP(r11)mflrr10stwr10,_LINK(r11)-mfctrr10-stwr10,_NIP(r11)mfcrr10rlwinmr10,r10,0,4,2/* Clear SO bit in CR */stwr10,_CCR(r11)/* save registers */
Userspace Execution protection and fast syscall entry were implemented
independently from each other and were both merged in kernel 5.2,
leading to syscall entry missing userspace execution protection.
On syscall entry, execution of user space memory must be
locked in the same way as on exception entry.
Fixes: b86fb88855ea ("powerpc/32: implement fast entry for syscalls on non BOOKE")
Cc: stable@vger.kernel.org
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/entry_32.S | 3 +++
1 file changed, 3 insertions(+)
@@ -29,6 +29,8 @@ notrace long system_call_exception(long r3, long r4, long r5,{syscall_fnf;+regs->orig_gpr3=r3;+if(IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))BUG_ON(irq_soft_mask_return()!=IRQS_ALL_DISABLED);
Instead of hard comparing task flags with _TIF_32BIT, use
is_compat_task(). The advantage is that it returns 0 on PPC32
allthough _TIF_32BIT is always set.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/syscall.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
@@ -116,7 +118,7 @@ notrace long system_call_exception(long r3, long r4, long r5,/* May be faster to do array_index_nospec? */barrier_nospec();-if(unlikely(is_32bit_task())){+if(unlikely(is_compat_task())){f=(void*)compat_sys_call_table[r0];r3&=0x00000000ffffffffULL;
@@ -39,7 +39,7 @@ notrace long system_call_exception(long r3, long r4, long r5,BUG_ON(!(regs->msr&MSR_RI));BUG_ON(!(regs->msr&MSR_PR));BUG_ON(!FULL_REGS(regs));-BUG_ON(regs->softe!=IRQS_ENABLED);+BUG_ON(arch_irq_disabled_regs(regs));#ifdef CONFIG_PPC_PKEYif(mmu_has_feature(MMU_FTR_PKEY)){
@@ -77,7 +77,7 @@ notrace long system_call_exception(long r3, long r4, long r5,*frame,oriftheunwinderwastaughtthefirststackframealways*returnstouserwithIRQS_ENABLED,thisstorecouldbeavoided!*/-regs->softe=IRQS_ENABLED;+irq_soft_mask_regs_set_state(regs,IRQS_ENABLED);local_irq_enable();
@@ -147,6 +147,7 @@ static notrace inline bool prep_irq_for_enabled_exit(bool clear_ri)__hard_EE_RI_disable();else__hard_irq_disable();+#ifdef CONFIG_PPC64if(unlikely(lazy_irq_pending_nocheck())){/* Took an interrupt, may have more exit work to do. */if(clear_ri)
@@ -281,7 +282,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,returnret;}-#ifdef CONFIG_PPC_BOOK3S /* BOOK3E not yet using this */+#ifdef CONFIG_PPC_BOOK3S_64 /* BOOK3E not yet using this */notraceunsignedlonginterrupt_exit_user_prepare(structpt_regs*regs,unsignedlongmsr){#ifdef CONFIG_PPC_BOOK3E
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-01-26 11:35:45
Excerpts from Christophe Leroy's message of January 26, 2021 12:48 am:
quoted hunk
Only PPC64 has scv. No need to check the 0x7ff0 trap on PPC32.
And ignore the scv parameter in syscall_exit_prepare (Save 14 cycles
346 => 332 cycles)
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/entry_32.S | 1 -
arch/powerpc/kernel/syscall.c | 7 +++++--
2 files changed, 5 insertions(+), 3 deletions(-)
@@ -86,7 +86,7 @@ notrace long system_call_exception(long r3, long r4, long r5,local_irq_enable();if(unlikely(current_thread_info()->flags&_TIF_SYSCALL_DOTRACE)){-if(unlikely(regs->trap==0x7ff0)){+if(IS_ENABLED(CONFIG_PPC64)&&unlikely(regs->trap==0x7ff0)){/* Unsupported scv vector */_exception(SIGILL,regs,ILL_ILLOPC,regs->nip);returnregs->gpr[3];
@@ -109,7 +109,7 @@ notrace long system_call_exception(long r3, long r4, long r5,r8=regs->gpr[8];}elseif(unlikely(r0>=NR_syscalls)){-if(unlikely(regs->trap==0x7ff0)){+if(IS_ENABLED(CONFIG_PPC64)&&unlikely(regs->trap==0x7ff0)){
Perhaps this could be hidden behind a function like trap_is_scv()?
trap_is_unsupported_scv() ?
Thanks,
Nick
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-01-26 11:38:01
Excerpts from Christophe Leroy's message of January 26, 2021 12:48 am:
syscall_64.c will be reused almost as is for PPC32.
Rename it syscall.c
Could you rename it to interrupt.c instead? A system call is an
interrupt, and the file now also has code to return from other
interrupts as well, and it matches the new asm/interrupt.h from
the interrupts series.
Thanks,
Nick
From: David Laight <hidden> Date: 2021-01-26 11:38:52
From: Nicholas Piggin
Sent: 26 January 2021 10:21
Excerpts from Christophe Leroy's message of January 26, 2021 12:48 am:
quoted
syscall_64.c will be reused almost as is for PPC32.
Rename it syscall.c
Could you rename it to interrupt.c instead? A system call is an
interrupt, and the file now also has code to return from other
interrupts as well, and it matches the new asm/interrupt.h from
the interrupts series.
Hmmm....
That might make it harder for someone looking for the system call
entry code to find it.
In some sense interrupts are the simpler case.
Especially when comparing with other architectures which have
special instructions for syscall entry.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
@@ -29,6 +29,8 @@ notrace long system_call_exception(long r3, long r4, long r5,{syscall_fnf;+regs->orig_gpr3=r3;+if(IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))BUG_ON(irq_soft_mask_return()!=IRQS_ALL_DISABLED);
@@ -115,6 +115,9 @@ notrace long system_call_exception(long r3, long r4, long r5,returnregs->gpr[3];}return-ENOSYS;+}else{+/* Restore r3 from orig_gpr3 to free up a volatile reg */+r3=regs->orig_gpr3;}/* May be faster to do array_index_nospec? */
--
Nice optimisation, great analysis and catch. I'll have to test it on
ppc64.
Thanks,
Nick
In preparation of porting PPC32 to C syscall entry/exit,
rewrite the following helpers as static inline functions and
add support for PPC32 in them:
__hard_irq_enable()
__hard_irq_disable()
__hard_EE_RI_disable()
__hard_RI_enable()
Then use them in PPC32 version of arch_local_irq_disable()
and arch_local_irq_enable() to avoid code duplication.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/hw_irq.h | 75 +++++++++++++++++++++----------
arch/powerpc/include/asm/reg.h | 1 +
2 files changed, 52 insertions(+), 24 deletions(-)
In system_call_exception(), MSR_RI also needs to be checked on 8xx.
Only booke and 40x doesn't have MSR_RI.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/syscall.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
@@ -39,7 +39,7 @@ notrace long system_call_exception(long r3, long r4, long r5,trace_hardirqs_off();/* finish reconciling */-if(IS_ENABLED(CONFIG_PPC_BOOK3S))+if(!IS_ENABLED(CONFIG_BOOKE)&&!IS_ENABLED(CONFIG_40x))BUG_ON(!(regs->msr&MSR_RI));BUG_ON(!(regs->msr&MSR_PR));BUG_ON(!FULL_REGS(regs));
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-01-27 23:51:09
Excerpts from David Laight's message of January 26, 2021 8:28 pm:
From: Nicholas Piggin
quoted
Sent: 26 January 2021 10:21
Excerpts from Christophe Leroy's message of January 26, 2021 12:48 am:
quoted
syscall_64.c will be reused almost as is for PPC32.
Rename it syscall.c
Could you rename it to interrupt.c instead? A system call is an
interrupt, and the file now also has code to return from other
interrupts as well, and it matches the new asm/interrupt.h from
the interrupts series.
Hmmm....
That might make it harder for someone looking for the system call
entry code to find it.
It's very grep'able.
In some sense interrupts are the simpler case.
Especially when comparing with other architectures which have
special instructions for syscall entry.
powerpc does have a special instruction for syscall, and it causes a
system call interrupt.
I'm not sure about other architectures, but for powerpc its more
sensible to call it interrupt.c than syscall.c.
Thanks,
Nick
Excerpts from David Laight's message of January 26, 2021 8:28 pm:
quoted
From: Nicholas Piggin
quoted
Sent: 26 January 2021 10:21
Excerpts from Christophe Leroy's message of January 26, 2021 12:48 am:
quoted
syscall_64.c will be reused almost as is for PPC32.
Rename it syscall.c
Could you rename it to interrupt.c instead? A system call is an
interrupt, and the file now also has code to return from other
interrupts as well, and it matches the new asm/interrupt.h from
the interrupts series.
Hmmm....
That might make it harder for someone looking for the system call
entry code to find it.
It's very grep'able.
quoted
In some sense interrupts are the simpler case.
Especially when comparing with other architectures which have
special instructions for syscall entry.
powerpc does have a special instruction for syscall, and it causes a
system call interrupt.
I'm not sure about other architectures, but for powerpc its more
sensible to call it interrupt.c than syscall.c.
Many other architectures have a syscall.c but for a different purpose: it contains arch specific
system calls. We have that in powerpc as well, it is called syscalls.c
So to avoid confusion, I'll rename it. But I think "interrupt" is maybe not the right name. An
interrupt most of the time refers to IRQ. For me system call is not an interrupt in the way it
doesn't unexpectedly interrupt a program flow. In powerpc manuals it is generally called exceptions,
no I'm more inclined to call it exception.c
Christophe
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-02-02 06:39:38
Excerpts from Christophe Leroy's message of February 2, 2021 4:15 pm:
Le 28/01/2021 à 00:50, Nicholas Piggin a écrit :
quoted
Excerpts from David Laight's message of January 26, 2021 8:28 pm:
quoted
From: Nicholas Piggin
quoted
Sent: 26 January 2021 10:21
Excerpts from Christophe Leroy's message of January 26, 2021 12:48 am:
quoted
syscall_64.c will be reused almost as is for PPC32.
Rename it syscall.c
Could you rename it to interrupt.c instead? A system call is an
interrupt, and the file now also has code to return from other
interrupts as well, and it matches the new asm/interrupt.h from
the interrupts series.
Hmmm....
That might make it harder for someone looking for the system call
entry code to find it.
It's very grep'able.
quoted
In some sense interrupts are the simpler case.
Especially when comparing with other architectures which have
special instructions for syscall entry.
powerpc does have a special instruction for syscall, and it causes a
system call interrupt.
I'm not sure about other architectures, but for powerpc its more
sensible to call it interrupt.c than syscall.c.
Many other architectures have a syscall.c but for a different purpose: it contains arch specific
system calls. We have that in powerpc as well, it is called syscalls.c
So to avoid confusion, I'll rename it. But I think "interrupt" is maybe not the right name. An
interrupt most of the time refers to IRQ.
That depends what you mean by interrupt and IRQ.
Linux kind of considers any asynchronous maskable interrupt an irq
(local_irq_disable()). But if you say irq it's more likely to mean
a device interrupt, and "interrupt" usually refres to the asynch
ones.
But Linux doesn't really assign names to synchronous interrupts in
core code. It doesn't say they aren't interrupts, it just doesn't
really have a convention for them at all.
Other architectures e.g., x86 also have things like interrupt
descriptor table for synchronous interrupts as well. That's where
I got the interrupt wrappers code from actually.
So it's really fine to use the proper arch-specific names for things
in arch code. I'm trying to slowly change names from exception to
interrupt.
For me system call is not an interrupt in the way it
doesn't unexpectedly interrupt a program flow. In powerpc manuals it is generally called exceptions,
no I'm more inclined to call it exception.c
Actually that's backwards. Powerpc manuals (at least the one I look at)
calls them all interrupts including system calls, and also the system
call interrupt is actually the only one that doesn't appear to be
associated with an exception.
Also there is no distinction about expecte/unexpected -- a data storage
interrupt is expected if you access a location without the right access
permissions for example, but it is still an interrupt.
These handlers very specifically deal with the change to execution flow
(i.e., the interrupt), they do *not* deal with the exception which may
be associated with it (that is the job of the handler).
And on the other hand you can deal with exceptions in some cases without
taking an interrupt at all. For example if you had MSR[EE]=0 you could
change the decrementer or execute msgclr or change HMER SPR etc to clear
various exceptions without ever taking the interrupt.
Thanks,
Nick
Excerpts from Christophe Leroy's message of February 2, 2021 4:15 pm:
quoted
Le 28/01/2021 à 00:50, Nicholas Piggin a écrit :
quoted
Excerpts from David Laight's message of January 26, 2021 8:28 pm:
quoted
From: Nicholas Piggin
quoted
Sent: 26 January 2021 10:21
Excerpts from Christophe Leroy's message of January 26, 2021 12:48 am:
quoted
syscall_64.c will be reused almost as is for PPC32.
Rename it syscall.c
Could you rename it to interrupt.c instead? A system call is an
interrupt, and the file now also has code to return from other
interrupts as well, and it matches the new asm/interrupt.h from
the interrupts series.
Hmmm....
That might make it harder for someone looking for the system call
entry code to find it.
It's very grep'able.
quoted
In some sense interrupts are the simpler case.
Especially when comparing with other architectures which have
special instructions for syscall entry.
powerpc does have a special instruction for syscall, and it causes a
system call interrupt.
I'm not sure about other architectures, but for powerpc its more
sensible to call it interrupt.c than syscall.c.
Many other architectures have a syscall.c but for a different purpose: it contains arch specific
system calls. We have that in powerpc as well, it is called syscalls.c
So to avoid confusion, I'll rename it. But I think "interrupt" is maybe not the right name. An
interrupt most of the time refers to IRQ.
That depends what you mean by interrupt and IRQ.
Linux kind of considers any asynchronous maskable interrupt an irq
(local_irq_disable()). But if you say irq it's more likely to mean
a device interrupt, and "interrupt" usually refres to the asynch
ones.
But Linux doesn't really assign names to synchronous interrupts in
core code. It doesn't say they aren't interrupts, it just doesn't
really have a convention for them at all.
Other architectures e.g., x86 also have things like interrupt
descriptor table for synchronous interrupts as well. That's where
I got the interrupt wrappers code from actually.
So it's really fine to use the proper arch-specific names for things
in arch code. I'm trying to slowly change names from exception to
interrupt.
quoted
For me system call is not an interrupt in the way it
doesn't unexpectedly interrupt a program flow. In powerpc manuals it is generally called exceptions,
no I'm more inclined to call it exception.c
Actually that's backwards. Powerpc manuals (at least the one I look at)
calls them all interrupts including system calls, and also the system
call interrupt is actually the only one that doesn't appear to be
associated with an exception.
Also there is no distinction about expecte/unexpected -- a data storage
interrupt is expected if you access a location without the right access
permissions for example, but it is still an interrupt.
These handlers very specifically deal with the change to execution flow
(i.e., the interrupt), they do *not* deal with the exception which may
be associated with it (that is the job of the handler).
And on the other hand you can deal with exceptions in some cases without
taking an interrupt at all. For example if you had MSR[EE]=0 you could
change the decrementer or execute msgclr or change HMER SPR etc to clear
various exceptions without ever taking the interrupt.
Ok, let's call it interrupt.c then, to be consistant with the interrupt wrapper story.
Christophe
On Tue, Feb 02, 2021 at 04:38:31PM +1000, Nicholas Piggin wrote:
quoted
So to avoid confusion, I'll rename it. But I think "interrupt" is maybe not the right name. An
interrupt most of the time refers to IRQ.
That depends what you mean by interrupt and IRQ.
In the PowerPC architecture, an exception is an abnormal condition, and
that can often cause an interrupt. What Christophe colloquially calls
an "IRQ" here is called an external exception c.q. external interrupt.
But if you say irq it's more likely to mean
a device interrupt, and "interrupt" usually refres to the asynch
ones.
Power talks about "instruction-caused interrupts", for one aspect of the
difference here; and "precise" / "imprecise" interrupts for another.
So it's really fine to use the proper arch-specific names for things
in arch code. I'm trying to slowly change names from exception to
interrupt.
Thanks :-)
quoted
For me system call is not an interrupt in the way it
doesn't unexpectedly interrupt a program flow. In powerpc manuals it is generally called exceptions,
no I'm more inclined to call it exception.c
Actually that's backwards. Powerpc manuals (at least the one I look at)
calls them all interrupts including system calls, and also the system
call interrupt is actually the only one that doesn't appear to be
associated with an exception.
Yeah. You could easily make such an exception, which is set when you
execute a system call instruction, and cleared when the interrupt is
taken, of course; but the architecture doesn't.
And on the other hand you can deal with exceptions in some cases without
taking an interrupt at all. For example if you had MSR[EE]=0 you could
change the decrementer or execute msgclr or change HMER SPR etc to clear
various exceptions without ever taking the interrupt.
A well-known example is the exception bits in the FPSCR, which do not
cause an interrupt unless the corresponding enable bits are also set.
Segher
Excerpts from Christophe Leroy's message of January 26, 2021 12:48 am:
quoted
syscall_64.c will be reused almost as is for PPC32.
Rename it syscall.c
Could you rename it to interrupt.c instead? A system call is an
interrupt, and the file now also has code to return from other
interrupts as well, and it matches the new asm/interrupt.h from
the interrupts series.
Excerpts from Christophe Leroy's message of January 26, 2021 12:48 am:
quoted
Only PPC64 has scv. No need to check the 0x7ff0 trap on PPC32.
And ignore the scv parameter in syscall_exit_prepare (Save 14 cycles
346 => 332 cycles)
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/entry_32.S | 1 -
arch/powerpc/kernel/syscall.c | 7 +++++--
2 files changed, 5 insertions(+), 3 deletions(-)
@@ -86,7 +86,7 @@ notrace long system_call_exception(long r3, long r4, long r5,local_irq_enable();if(unlikely(current_thread_info()->flags&_TIF_SYSCALL_DOTRACE)){-if(unlikely(regs->trap==0x7ff0)){+if(IS_ENABLED(CONFIG_PPC64)&&unlikely(regs->trap==0x7ff0)){/* Unsupported scv vector */_exception(SIGILL,regs,ILL_ILLOPC,regs->nip);returnregs->gpr[3];
@@ -109,7 +109,7 @@ notrace long system_call_exception(long r3, long r4, long r5,r8=regs->gpr[8];}elseif(unlikely(r0>=NR_syscalls)){-if(unlikely(regs->trap==0x7ff0)){+if(IS_ENABLED(CONFIG_PPC64)&&unlikely(regs->trap==0x7ff0)){
Perhaps this could be hidden behind a function like trap_is_scv()?
trap_is_unsupported_scv() ?