In the kernel we do follow the below sequence in different code paths.
pte = ptep_get_clear(ptep)
....
set_pte_at(ptep, pte)
We do that for mremap, autonuma protection update and softdirty clearing. This
implies our optimization to skip a tlb flush when clearing a pte update is
not valid, because for DD1 system that followup set_pte_at will be done witout
doing the required tlbflush. Fix that by always doing the dd1 style pte update
irrespective of new_pte value. In a later patch we will optimize the application
exit case.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Aneesh Kumar K.V <redacted>
---
arch/powerpc/include/asm/book3s/64/radix.h | 12 +++---------
1 file changed, 3 insertions(+), 9 deletions(-)
This helps us to do some optimization for application exit case, where we can
skip the DD1 style pte update sequence.
Signed-off-by: Aneesh Kumar K.V <redacted>
---
arch/powerpc/include/asm/book3s/64/pgtable.h | 17 +++++++++++++++++
arch/powerpc/include/asm/book3s/64/radix.h | 23 ++++++++++++++++++++++-
2 files changed, 39 insertions(+), 1 deletion(-)
We do them at the start of tlb flush, and we are sure a pte update will be
followed by a tlbflush. Hence we can skip the ptesync in pte update helpers.
Signed-off-by: Aneesh Kumar K.V <redacted>
---
arch/powerpc/include/asm/book3s/64/radix.h | 3 ---
1 file changed, 3 deletions(-)
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2017-02-09 03:50:01
On Thu, 2017-02-09 at 08:28 +0530, Aneesh Kumar K.V wrote:
In the kernel we do follow the below sequence in different code paths.
pte = ptep_get_clear(ptep)
....
set_pte_at(ptep, pte)
We do that for mremap, autonuma protection update and softdirty clearing. This
implies our optimization to skip a tlb flush when clearing a pte update is
not valid, because for DD1 system that followup set_pte_at will be done witout
doing the required tlbflush. Fix that by always doing the dd1 style pte update
irrespective of new_pte value. In a later patch we will optimize the application
exit case.
What about my change to set_pte_at() ? We seem to be overwriting valid PTEs,
shouldn't we deal with that ?
Cheers,
Ben.
quoted hunk
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
* new value of pte
*/
new_pte = (old_pte | set) & ~clr;
- /*
- * If we are trying to clear the pte, we can skip
- * the below sequence and batch the tlb flush. The
- * tlb flush batching is done by mmu gather code
- */
- if (new_pte) {
- asm volatile("ptesync" : : : "memory");
- radix__flush_tlb_pte_p9_dd1(old_pte, mm, addr);
+ asm volatile("ptesync" : : : "memory");
+ radix__flush_tlb_pte_p9_dd1(old_pte, mm, addr);
+ if (new_pte)
__radix_pte_update(ptep, 0, new_pte);
- }
} else
old_pte = __radix_pte_update(ptep, clr, set);
asm volatile("ptesync" : : : "memory");
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2017-02-09 04:15:41
On Thu, 2017-02-09 at 14:49 +1100, Benjamin Herrenschmidt wrote:
On Thu, 2017-02-09 at 08:28 +0530, Aneesh Kumar K.V wrote:
quoted
In the kernel we do follow the below sequence in different code
paths.
pte = ptep_get_clear(ptep)
....
set_pte_at(ptep, pte)
We do that for mremap, autonuma protection update and softdirty
clearing. This
implies our optimization to skip a tlb flush when clearing a pte
update is
not valid, because for DD1 system that followup set_pte_at will be
done witout
doing the required tlbflush. Fix that by always doing the dd1 style
pte update
irrespective of new_pte value. In a later patch we will optimize
the application
exit case.
What about my change to set_pte_at() ? We seem to be overwriting
valid PTEs,
shouldn't we deal with that ?
So the HW guys confirmed that the TLB will never cache a valid entry
that has all permissions clear. That leaves the THP write problem
though.
Cheers,
Ben.
Cheers,
Ben.
quoted
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
* new value of pte
*/
new_pte = (old_pte | set) & ~clr;
- /*
- * If we are trying to clear the pte, we can
skip
- * the below sequence and batch the tlb flush.
The
- * tlb flush batching is done by mmu gather code
- */
- if (new_pte) {
- asm volatile("ptesync" : : : "memory");
- radix__flush_tlb_pte_p9_dd1(old_pte, mm,
addr);
+ asm volatile("ptesync" : : : "memory");
+ radix__flush_tlb_pte_p9_dd1(old_pte, mm, addr);
+ if (new_pte)
__radix_pte_update(ptep, 0, new_pte);
- }
} else
old_pte = __radix_pte_update(ptep, clr, set);
asm volatile("ptesync" : : : "memory");
On Thu, 2017-02-09 at 14:49 +1100, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2017-02-09 at 08:28 +0530, Aneesh Kumar K.V wrote:
quoted
In the kernel we do follow the below sequence in different code
paths.
pte = ptep_get_clear(ptep)
....
set_pte_at(ptep, pte)
We do that for mremap, autonuma protection update and softdirty
clearing. This
implies our optimization to skip a tlb flush when clearing a pte
update is
not valid, because for DD1 system that followup set_pte_at will be
done witout
doing the required tlbflush. Fix that by always doing the dd1 style
pte update
irrespective of new_pte value. In a later patch we will optimize
the application
exit case.
What about my change to set_pte_at() ? We seem to be overwriting
valid PTEs,
shouldn't we deal with that ?
So the HW guys confirmed that the TLB will never cache a valid entry
that has all permissions clear. That leaves the THP write problem
though.
From: Michael Neuling <hidden> Date: 2017-02-14 04:17:15
On Thu, 2017-02-09 at 08:28 +0530, Aneesh Kumar K.V wrote:
In the kernel we do follow the below sequence in different code paths.
pte =3D ptep_get_clear(ptep)
....
set_pte_at(ptep, pte)
=20
We do that for mremap, autonuma protection update and softdirty clearing.=
This
implies our optimization to skip a tlb flush when clearing a pte update i=
s
not valid, because for DD1 system that followup set_pte_at will be done w=
itout
doing the required tlbflush. Fix that by always doing the dd1 style pte u=
pdate
irrespective of new_pte value. In a later patch we will optimize the
application
exit case.
=20
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Aneesh Kumar K.V <redacted>
@@ -139,7 +139,7 @@ static inline unsigned long radix__pte_update(struct
mm_struct *mm,
=C2=A0
=C2=A0 unsigned long new_pte;
=C2=A0
- old_pte =3D __radix_pte_update(ptep, ~0, 0);
+ old_pte =3D __radix_pte_update(ptep, ~0ul, 0);
=C2=A0 /*
=C2=A0 =C2=A0* new value of pte
=C2=A0 =C2=A0*/
@@ -157,6 +157,27 @@ static inline unsigned long radix__pte_update(struct
mm_struct *mm,
=C2=A0 return old_pte;
=C2=A0}
=C2=A0
+static inline pte_t radix__ptep_get_and_clear_full(struct mm_struct *mm,
+ =C2=A0=C2=A0=C2=A0unsigned long addr,
+ =C2=A0=C2=A0=C2=A0pte_t *ptep, int full)
+{
+ unsigned long old_pte;
+
+ if (full) {
+ /*
+ =C2=A0* If we are trying to clear the pte, we can skip
+ =C2=A0* the DD1 pte update sequence and batch the tlb flush. The
+ =C2=A0* tlb flush batching is done by mmu gather code. We
+ =C2=A0* still keep the cmp_xchg update to make sure we get
+ =C2=A0* correct R/C bit which might be updated via Nest MMU.
+ =C2=A0*/
+ old_pte =3D __radix_pte_update(ptep, ~0ul, 0);
+ } else
+ old_pte =3D radix__pte_update(mm, addr, ptep, ~0ul, 0, 0);
+
+ return __pte(old_pte);
+}
+
=C2=A0/*
=C2=A0 * Set the dirty and/or accessed bits atomically in a linux PTE, th=
From: Michael Ellerman <hidden> Date: 2017-02-16 05:59:47
On Thu, 2017-02-09 at 02:58:19 UTC, "Aneesh Kumar K.V" wrote:
In the kernel we do follow the below sequence in different code paths.
pte = ptep_get_clear(ptep)
....
set_pte_at(ptep, pte)
We do that for mremap, autonuma protection update and softdirty clearing. This
implies our optimization to skip a tlb flush when clearing a pte update is
not valid, because for DD1 system that followup set_pte_at will be done witout
doing the required tlbflush. Fix that by always doing the dd1 style pte update
irrespective of new_pte value. In a later patch we will optimize the application
exit case.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Aneesh Kumar K.V <redacted>
Tested-by: Michael Neuling <redacted>