Re: [PATCH v2] powerpc: Don't clobber fr0/vs0 during fp|altivec register save
From: Timothy Pearson <tpearson@raptorengineering.com>
Date: 2023-11-24 00:03:01
Also in:
regressions
Subsystem:
linux for powerpc (32-bit and 64-bit), the rest · Maintainers:
Madhavan Srinivasan, Michael Ellerman, Linus Torvalds
----- Original Message -----
From: "Michael Ellerman" <mpe@ellerman.id.au> To: "Timothy Pearson" <tpearson@raptorengineering.com> Cc: "Jens Axboe" <axboe@kernel.dk>, "regressions" <redacted>, "npiggin" <npiggin@gmail.com>, "christophe leroy" [off-list ref], "linuxppc-dev" [off-list ref] Sent: Tuesday, November 21, 2023 11:01:50 PM Subject: Re: [PATCH v2] powerpc: Don't clobber fr0/vs0 during fp|altivec register save
Timothy Pearson [off-list ref] writes:quoted
...quoted
So a little more detail on this, just to put it to rest properly vs. assuming hand analysis caught every possible pathway. :) The debugging that generates this stack trace also verifies the following in __giveup_fpu(): 1.) tsk->thread.fp_state.fpr doesn't contain the FPSCR contents prior to calling save_fpu() 2.) tsk->thread.fp_state.fpr contains the FPSCR contents directly after calling save_fpu() 3.) MSR_FP is set both in the task struct and in the live MSR. Only if all three conditions are met will it generate the trace. This is a generalization of the hack I used to find the problem in the first place. If the state will subsequently be reloaded from the thread struct, that means we're reloading the registers from the thread struct that we just verified was corrupted by the earlier save_fpu() call. There are only two ways I can see for that to be true -- one is if the registers were already clobbered when giveup_all() was entered, and the other is if save_fpu() went ahead and clobbered them right here inside giveup_all(). To see which scenario we were dealing with, I added a bit more instrumentation to dump the current register state if MSR_FP bit was already set in registers (i.e. not dumping data from task struct, but using the live FPU registers instead), and sure enough the registers are corrupt on entry, so something else has already called save_fpu() before we even hit giveup_all() in this call chain.Can you share the debug patch you're using? cheers
Sure, here you go. Note that with my FPU patch there is no WARN_ON hit, at least in my testing, so it isn't userspace purposefully loading the fr0/vs0 register with the FPSCR.
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index 392404688cec..bde57dc3262a 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c@@ -154,7 +154,49 @@ static void __giveup_fpu(struct task_struct *tsk) { unsigned long msr; + // DEBUGGING + uint64_t prev_fpr0 = *(((uint64_t*)(&tsk->thread.fp_state.fpr[0]))+0); + uint64_t prev_fpr1 = *(((uint64_t*)(&tsk->thread.fp_state.fpr[0]))+1); + struct thread_fp_state debug_fp_state; + unsigned long currentmsr = mfmsr(); + + if (currentmsr & MSR_FP) { + store_fp_state(&debug_fp_state); + load_fp_state(&debug_fp_state); + } + save_fpu(tsk); + + // DEBUGGING + if (tsk->thread.regs->msr & MSR_FP) { + if (((*(((uint64_t*)(&tsk->thread.fp_state.fpr[0]))+0) == 0x82004000) && (prev_fpr0 != 0x82004000)) + || ((*(((uint64_t*)(&tsk->thread.fp_state.fpr[0]))+1) == 0x82004000) && (prev_fpr1 != 0x82004000))) + { + WARN_ON(1); + + printk("[TS %lld] In __giveup_fpu() for process [comm: '%s' pid %d tid %d], before save current " + "fp0: 0x%016llx/%016llx fp1: 0x%016llx/%016llx fp8: 0x%016llx/%016llx fp9: 0x%016llx/%016llx" + " msr: 0x%016lx (FP %d VSX %d EE %d) on core %d\n", + ktime_get_boottime_ns(), current->comm, current->pid, current->tgid, + *(((uint64_t*)(&debug_fp_state.fpr[0]))+0), *(((uint64_t*)(&debug_fp_state.fpr[0]))+1), + *(((uint64_t*)(&debug_fp_state.fpr[1]))+0), *(((uint64_t*)(&debug_fp_state.fpr[1]))+1), + *(((uint64_t*)(&debug_fp_state.fpr[8]))+0), *(((uint64_t*)(&debug_fp_state.fpr[8]))+1), + *(((uint64_t*)(&tsk->thread.fp_state.fpr[9]))+0), *(((uint64_t*)(&tsk->thread.fp_state.fpr[9]))+1), + currentmsr, !!(currentmsr & MSR_FP), !!(currentmsr & MSR_VSX), !!(currentmsr & MSR_EE), raw_smp_processor_id()); + + printk("[TS %lld] In __giveup_fpu() for process [comm: '%s' pid %d tid %d], after save saved " + "fp0: 0x%016llx/%016llx fp1: 0x%016llx/%016llx fp8: 0x%016llx/%016llx fp9: 0x%016llx/%016llx" + " msr: 0x%016lx (FP %d VSX %d EE %d) on core %d\n", + ktime_get_boottime_ns(), current->comm, current->pid, current->tgid, + *(((uint64_t*)(&tsk->thread.fp_state.fpr[0]))+0), *(((uint64_t*)(&tsk->thread.fp_state.fpr[0]))+1), + *(((uint64_t*)(&tsk->thread.fp_state.fpr[1]))+0), *(((uint64_t*)(&tsk->thread.fp_state.fpr[1]))+1), + *(((uint64_t*)(&tsk->thread.fp_state.fpr[8]))+0), *(((uint64_t*)(&tsk->thread.fp_state.fpr[8]))+1), + *(((uint64_t*)(&tsk->thread.fp_state.fpr[9]))+0), *(((uint64_t*)(&tsk->thread.fp_state.fpr[9]))+1), + tsk->thread.regs->msr, !!(tsk->thread.regs->msr & MSR_FP), !!(tsk->thread.regs->msr & MSR_VSX), !!(tsk->thread.regs->msr & MSR_EE), raw_smp_processor_id()); + } + } + + msr = tsk->thread.regs->msr; msr &= ~(MSR_FP|MSR_FE0|MSR_FE1); if (cpu_has_feature(CPU_FTR_VSX))