From: Nicholas Piggin <npiggin@gmail.com> Date: 2023-05-24 06:09:20
In the process of doing patch 4, I found a few things we could improve
and tighten up with mm_cpumask handling, so added those first. They're
mostly just debugging, no real fixes or dependency on patch 4 there.
Thanks,
Nick
Nicholas Piggin (4):
powerpc: Account mm_cpumask and active_cpus in init_mm
powerpc/64s: Use dec_mm_active_cpus helper
powerpc: Add mm_cpumask warning when context switching
powerpc/64s/radix: combine final TLB flush and lazy tlb mm shootdown
IPIs
arch/powerpc/include/asm/book3s/64/mmu.h | 2 +-
arch/powerpc/include/asm/mmu_context.h | 1 +
arch/powerpc/kernel/setup-common.c | 6 ++++-
arch/powerpc/kernel/smp.c | 12 ++++++++++
arch/powerpc/mm/book3s64/radix_tlb.c | 28 ++++++++++++++++++++++--
arch/powerpc/mm/mmu_context.c | 8 +++++--
6 files changed, 51 insertions(+), 6 deletions(-)
--
2.40.1
From: Nicholas Piggin <npiggin@gmail.com> Date: 2023-05-24 06:10:07
init_mm mm_cpumask and context.active_cpus is not maintained at boot
and hotplug. This seems to be harmless because init_mm does not have a
userspace and so never gets user TLBs flushed, but it looks odd and it
prevents some sanity checks being added.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/setup-common.c | 6 +++++-
arch/powerpc/kernel/smp.c | 12 ++++++++++++
arch/powerpc/mm/mmu_context.c | 1 +
3 files changed, 18 insertions(+), 1 deletion(-)
@@ -969,8 +969,12 @@ void __init setup_arch(char **cmdline_p)klp_init_thread_info(&init_task);setup_initial_init_mm(_stext,_etext,_edata,_end);-+/* sched_init() does the mmgrab(&init_mm) for the primary CPU */+VM_WARN_ON(cpumask_test_cpu(smp_processor_id(),mm_cpumask(&init_mm)));+cpumask_set_cpu(smp_processor_id(),mm_cpumask(&init_mm));+inc_mm_active_cpus(&init_mm);mm_iommu_init(&init_mm);+irqstack_early_init();exc_lvl_early_init();emergency_stack_init();
@@ -47,6 +47,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,/* Mark this context has been used on the new CPU */if(!cpumask_test_cpu(smp_processor_id(),mm_cpumask(next))){+VM_WARN_ON_ONCE(next==&init_mm);cpumask_set_cpu(smp_processor_id(),mm_cpumask(next));inc_mm_active_cpus(next);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2023-05-24 06:10:55
Avoid open-coded atomic_dec on mm->context.active_cpus and use the
function made for it. Add CONFIG_DEBUG_VM underflow checking on the
counter.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/book3s/64/mmu.h | 2 +-
arch/powerpc/include/asm/mmu_context.h | 1 +
arch/powerpc/mm/book3s64/radix_tlb.c | 2 +-
3 files changed, 3 insertions(+), 2 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2023-05-24 06:11:43
When context switching away from an mm, add a CONFIG_DEBUG_VM warning
check to ensure this CPU is still set in the mask. This could catch
bugs where the mask is improperly trimmed while the CPU is still using
the mm.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/mmu_context.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
@@ -43,12 +43,13 @@ static inline void switch_mm_pgdir(struct task_struct *tsk,voidswitch_mm_irqs_off(structmm_struct*prev,structmm_struct*next,structtask_struct*tsk){+intcpu=smp_processor_id();boolnew_on_cpu=false;/* Mark this context has been used on the new CPU */-if(!cpumask_test_cpu(smp_processor_id(),mm_cpumask(next))){+if(!cpumask_test_cpu(cpu,mm_cpumask(next))){VM_WARN_ON_ONCE(next==&init_mm);-cpumask_set_cpu(smp_processor_id(),mm_cpumask(next));+cpumask_set_cpu(cpu,mm_cpumask(next));inc_mm_active_cpus(next);/*
From: Nicholas Piggin <npiggin@gmail.com> Date: 2023-05-24 06:12:30
This performs lazy tlb mm shootdown when doing the exit TLB flush when
all mm users go away and user mappings are removed, which avoids having
to do the lazy tlb mm shootdown IPIs on the final mmput when all kernel
references disappear.
powerpc/64s uses a broadcast TLBIE for the exit TLB flush if remote CPUs
need to be invalidated (unless TLBIE is disabled), so this doesn't
necessarily save IPIs but it does avoid a broadcast TLBIE which is quite
expensive.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/book3s64/radix_tlb.c | 26 +++++++++++++++++++++++++-
1 file changed, 25 insertions(+), 1 deletion(-)
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2023-07-18 02:55:28
Nicholas Piggin [off-list ref] writes:
This performs lazy tlb mm shootdown when doing the exit TLB flush when
all mm users go away and user mappings are removed, which avoids having
to do the lazy tlb mm shootdown IPIs on the final mmput when all kernel
references disappear.
powerpc/64s uses a broadcast TLBIE for the exit TLB flush if remote CPUs
need to be invalidated (unless TLBIE is disabled), so this doesn't
necessarily save IPIs but it does avoid a broadcast TLBIE which is quite
expensive.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/book3s64/radix_tlb.c | 26 +++++++++++++++++++++++++-
1 file changed, 25 insertions(+), 1 deletion(-)
On Tue Jul 18, 2023 at 12:54 PM AEST, Michael Ellerman wrote:
Nicholas Piggin [off-list ref] writes:
quoted
This performs lazy tlb mm shootdown when doing the exit TLB flush when
all mm users go away and user mappings are removed, which avoids having
to do the lazy tlb mm shootdown IPIs on the final mmput when all kernel
references disappear.
powerpc/64s uses a broadcast TLBIE for the exit TLB flush if remote CPUs
need to be invalidated (unless TLBIE is disabled), so this doesn't
necessarily save IPIs but it does avoid a broadcast TLBIE which is quite
expensive.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/book3s64/radix_tlb.c | 26 +++++++++++++++++++++++++-
1 file changed, 25 insertions(+), 1 deletion(-)
From: Michael Ellerman <hidden> Date: 2023-08-10 06:06:06
On Wed, 24 May 2023 16:08:17 +1000, Nicholas Piggin wrote:
In the process of doing patch 4, I found a few things we could improve
and tighten up with mm_cpumask handling, so added those first. They're
mostly just debugging, no real fixes or dependency on patch 4 there.
Thanks,
Nick
[...]
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2023-08-18 07:23:14
Nicholas Piggin [off-list ref] writes:
quoted hunk
When context switching away from an mm, add a CONFIG_DEBUG_VM warning
check to ensure this CPU is still set in the mask. This could catch
bugs where the mask is improperly trimmed while the CPU is still using
the mm.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/mm/mmu_context.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)