The main purpose of this big series is to:
- reorganise huge page handling to avoid using mm_slices.
- use huge pages to map kernel memory on the 8xx.
The 8xx supports 4 page sizes: 4k, 16k, 512k and 8M.
It uses 2 Level page tables, PGD having 1024 entries, each entry
covering 4M address space. Then each page table has 1024 entries.
At the time being, page sizes are managed in PGD entries, implying
the use of mm_slices as it can't mix several pages of the same size
in one page table.
The first purpose of this series is to reorganise things so that
standard page tables can also handle 512k pages. This is done by
adding a new _PAGE_HUGE flag which will be copied into the Level 1
entry in the TLB miss handler. That done, we have 2 types of pages:
- PGD entries to regular page tables handling 4k/16k and 512k pages
- PGD entries to hugepd tables handling 8M pages.
There is no need to mix 8M pages with other sizes, because a 8M page
will use more than what a single PGD covers.
Then comes the second purpose of this series. At the time being, the
8xx has implemented special handling in the TLB miss handlers in order
to transparently map kernel linear address space and the IMMR using
huge pages by building the TLB entries in assembly at the time of the
exception.
As mm_slices is only for user space pages, and also because it would
anyway not be convenient to slice kernel address space, it was not
possible to use huge pages for kernel address space. But after step
one of the series, it is now more flexible to use huge pages.
This series drop all assembly 'just in time' handling of huge pages
and use huge pages in page tables instead.
Once the above is done, then comes the cherry on cake:
- Use huge pages for KASAN shadow mapping
- Allow pinned TLBs with strict kernel rwx
- Allow pinned TLBs with debug pagealloc
Then, last but not least, those modifications for the 8xx allows the
following improvement on book3s/32:
- Mapping KASAN shadow with BATs
- Allowing BATs with debug pagealloc
All this allows to considerably simplify TLB miss handlers and associated
initialisation. The overhead of reading page tables is negligible
compared to the reduction of the miss handlers.
While we were at touching pte_update(), some cleanup was done
there too.
Tested widely on 8xx and 832x. Boot tested on QEMU MAC99.
Christophe Leroy (46):
powerpc/kasan: Fix shadow memory protection with CONFIG_KASAN_VMALLOC
powerpc/kasan: Fix error detection on memory allocation
powerpc/kasan: Fix issues by lowering KASAN_SHADOW_END
powerpc/kasan: Fix shadow pages allocation failure
powerpc/kasan: Remove unnecessary page table locking
powerpc/kasan: Refactor update of early shadow mappings
powerpc/kasan: Declare kasan_init_region() weak
powerpc/ptdump: Limit size of flags text to 1/2 chars on PPC32
powerpc/ptdump: Reorder flags
powerpc/ptdump: Add _PAGE_COHERENT flag
powerpc/ptdump: Display size of BATs
powerpc/ptdump: Standardise display of BAT flags
powerpc/ptdump: Properly handle non standard page size
powerpc/ptdump: Handle hugepd at PGD level
powerpc/32s: Don't warn when mapping RO data ROX.
powerpc/mm: Allocate static page tables for fixmap
powerpc/mm: Fix conditions to perform MMU specific management by
blocks on PPC32.
powerpc/mm: PTE_ATOMIC_UPDATES is only for 40x
powerpc/mm: Refactor pte_update() on nohash/32
powerpc/mm: Refactor pte_update() on book3s/32
powerpc/mm: Standardise __ptep_test_and_clear_young() params between
PPC32 and PPC64
powerpc/mm: Standardise pte_update() prototype between PPC32 and PPC64
powerpc/mm: Create a dedicated pte_update() for 8xx
powerpc/mm: Reduce hugepd size for 8M hugepages on 8xx
powerpc/8xx: Drop CONFIG_8xx_COPYBACK option
powerpc/8xx: Prepare handlers for _PAGE_HUGE for 512k pages.
powerpc/8xx: Manage 512k huge pages as standard pages.
powerpc/8xx: Only 8M pages are hugepte pages now
powerpc/8xx: MM_SLICE is not needed anymore
powerpc/8xx: Move PPC_PIN_TLB options into 8xx Kconfig
powerpc/8xx: Add function to update pinned TLBs
powerpc/8xx: Don't set IMMR map anymore at boot
powerpc/8xx: Always pin TLBs at startup.
powerpc/8xx: Drop special handling of Linear and IMMR mappings in I/D
TLB handlers
powerpc/8xx: Remove now unused TLB miss functions
powerpc/8xx: Move DTLB perf handling closer.
powerpc/mm: Don't be too strict with _etext alignment on PPC32
powerpc/8xx: Refactor kernel address boundary comparison
powerpc/8xx: Add a function to early map kernel via huge pages
powerpc/8xx: Map IMMR with a huge page
powerpc/8xx: Map linear memory with huge pages
powerpc/8xx: Allow STRICT_KERNEL_RwX with pinned TLB
powerpc/8xx: Allow large TLBs with DEBUG_PAGEALLOC
powerpc/8xx: Implement dedicated kasan_init_region()
powerpc/32s: Allow mapping with BATs with DEBUG_PAGEALLOC
powerpc/32s: Implement dedicated kasan_init_region()
arch/powerpc/Kconfig | 62 +---
arch/powerpc/configs/adder875_defconfig | 1 -
arch/powerpc/configs/ep88xc_defconfig | 1 -
arch/powerpc/configs/mpc866_ads_defconfig | 1 -
arch/powerpc/configs/mpc885_ads_defconfig | 1 -
arch/powerpc/configs/tqm8xx_defconfig | 1 -
arch/powerpc/include/asm/book3s/32/pgtable.h | 78 ++---
arch/powerpc/include/asm/fixmap.h | 4 +
arch/powerpc/include/asm/hugetlb.h | 6 +-
arch/powerpc/include/asm/kasan.h | 10 +-
.../include/asm/nohash/32/hugetlb-8xx.h | 32 +-
arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 75 +----
arch/powerpc/include/asm/nohash/32/pgtable.h | 104 +++----
arch/powerpc/include/asm/nohash/32/pte-8xx.h | 4 +-
arch/powerpc/include/asm/nohash/32/slice.h | 20 --
arch/powerpc/include/asm/nohash/64/pgtable.h | 28 +-
arch/powerpc/include/asm/nohash/pgtable.h | 2 +-
arch/powerpc/include/asm/pgtable.h | 2 +
arch/powerpc/include/asm/slice.h | 2 -
arch/powerpc/kernel/head_8xx.S | 292 ++++++------------
arch/powerpc/kernel/setup_32.c | 2 +-
arch/powerpc/kernel/vmlinux.lds.S | 3 +-
arch/powerpc/mm/book3s32/mmu.c | 12 +-
arch/powerpc/mm/hugetlbpage.c | 43 +--
arch/powerpc/mm/init_32.c | 12 +-
arch/powerpc/mm/kasan/8xx.c | 74 +++++
arch/powerpc/mm/kasan/Makefile | 2 +
arch/powerpc/mm/kasan/book3s_32.c | 57 ++++
arch/powerpc/mm/kasan/kasan_init_32.c | 91 +++---
arch/powerpc/mm/mmu_decl.h | 4 +
arch/powerpc/mm/nohash/8xx.c | 250 ++++++++-------
arch/powerpc/mm/pgtable.c | 34 +-
arch/powerpc/mm/pgtable_32.c | 22 +-
arch/powerpc/mm/ptdump/8xx.c | 52 ++--
arch/powerpc/mm/ptdump/bats.c | 41 ++-
arch/powerpc/mm/ptdump/ptdump.c | 72 +++--
arch/powerpc/mm/ptdump/ptdump.h | 2 +
arch/powerpc/mm/ptdump/shared.c | 58 ++--
arch/powerpc/perf/8xx-pmu.c | 10 -
arch/powerpc/platforms/8xx/Kconfig | 50 ++-
arch/powerpc/platforms/Kconfig.cputype | 2 +-
arch/powerpc/sysdev/cpm_common.c | 2 +
42 files changed, 820 insertions(+), 801 deletions(-)
delete mode 100644 arch/powerpc/include/asm/nohash/32/slice.h
create mode 100644 arch/powerpc/mm/kasan/8xx.c
create mode 100644 arch/powerpc/mm/kasan/book3s_32.c
--
2.25.0
With CONFIG_KASAN_VMALLOC, new page tables are created at the time
shadow memory for vmalloc area in unmapped. If some parts of the
page table still has entries to the zero page shadow memory, the
entries are wrongly marked RW.
With CONFIG_KASAN_VMALLOC, almost the entire kernel address space
is managed by KASAN. To make it simple, just create KASAN page tables
for the entire kernel space at kasan_init(). That doesn't use much
more space, and that's anyway already done for hash platforms.
Fixes: 3d4247fcc938 ("powerpc/32: Add support of KASAN_VMALLOC")
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/mm/kasan/kasan_init_32.c | 9 ++-------
1 file changed, 2 insertions(+), 7 deletions(-)
At the time being, KASAN_SHADOW_END is 0x100000000, which
is 0 in 32 bits representation.
This leads to a couple of issues:
- kasan_remap_early_shadow_ro() does nothing because the comparison
k_cur < k_end is always false.
- In ptdump, address comparison for markers display fails and the
marker's name is printed at the start of the KASAN area instead of
being printed at the end.
However, there is no need to shadow the KASAN shadow area itself,
so the KASAN shadow area can stop shadowing memory at the start
of itself.
With a PAGE_OFFSET set to 0xc0000000, KASAN shadow area is then going
from 0xf8000000 to 0xff000000.
Signed-off-by: Christophe Leroy <redacted>
Fixes: cbd18991e24f ("powerpc/mm: Fix an Oops in kasan_mmu_init()")
Cc: stable@vger.kernel.org
---
arch/powerpc/include/asm/kasan.h | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
Doing kasan pages allocation in MMU_init is too early, kernel doesn't
have access yet to the entire memory space and memblock_alloc() fails
when the kernel is a bit big.
Do it from kasan_init() instead.
Fixes: 2edb16efc899 ("powerpc/32: Add KASAN support")
Cc: stable@vger.kernel.org
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/kasan.h | 2 --
arch/powerpc/mm/init_32.c | 2 --
arch/powerpc/mm/kasan/kasan_init_32.c | 4 +++-
3 files changed, 3 insertions(+), 5 deletions(-)
@@ -170,8 +170,6 @@ void __init MMU_init(void)btext_unmap();#endif-kasan_mmu_init();-setup_kup();/* Shortly after that, the entire linear mapping will be available */
kasan_remap_early_shadow_ro() and kasan_unmap_early_shadow_vmalloc()
are both updating the early shadow mapping: the first one sets
the mapping read-only while the other clears the mapping.
Refactor and create kasan_update_early_region()
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/mm/kasan/kasan_init_32.c | 39 +++++++++++++--------------
1 file changed, 18 insertions(+), 21 deletions(-)
In order to alloc sub-arches to alloc KASAN regions using optimised
methods (Huge pages on 8xx, BATs on BOOK3S, ...), declare
kasan_init_region() weak.
Also make kasan_init_shadow_page_tables() accessible from outside,
so that it can be called from the specific kasan_init_region()
functions if needed.
And populate remaining KASAN address space only once performed
the region mapping, to allow 8xx to allocate hugepd instead of
standard page tables for mapping via 8M hugepages.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/kasan.h | 3 +++
arch/powerpc/mm/kasan/kasan_init_32.c | 21 +++++++++++----------
2 files changed, 14 insertions(+), 10 deletions(-)
@@ -112,6 +112,19 @@ static struct addr_marker address_markers[] = {seq_putc(m,c);\})+voidpt_dump_size(structseq_file*m,unsignedlongsize)+{+staticconstcharunits[]="KMGTPE";+constchar*unit=units;++/* Work out what appropriate unit to use */+while(!(size&1023)&&unit[1]){+size>>=10;+unit++;+}+pt_dump_seq_printf(m,"%9lu%c ",size,*unit);+}+staticvoiddump_flag_info(structpg_state*st,conststructflag_info*flag,u64pte,intnum){
@@ -164,13 +175,7 @@ static void dump_addr(struct pg_state *st, unsigned long addr)pt_dump_seq_printf(st->seq," "REG" ",st->start_pa);delta=(addr-st->start_address)>>10;}-/* Work out what appropriate unit to use */-while(!(delta&1023)&&unit[1]){-delta>>=10;-unit++;-}-pt_dump_seq_printf(st->seq,"%9lu%c",delta,*unit);-+pt_dump_size(st->seq,delta);}staticvoidnote_prot_wx(structpg_state*st,unsignedlongaddr)
For platforms using shared.c (4xx, Book3e, Book3s/32),
also handle the _PAGE_COHERENT flag with corresponds to the
M bit of the WIMG flags.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/mm/ptdump/shared.c | 5 +++++
1 file changed, 5 insertions(+)
Display BAT flags the same way as page flags: rwx and wimg
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/mm/ptdump/bats.c | 37 ++++++++++++++---------------------
1 file changed, 15 insertions(+), 22 deletions(-)
Setting init mem to NX shall depend on sinittext being mapped by
block, not on stext being mapped by block.
Setting text and rodata to RO shall depend on stext being mapped by
block, not on sinittext being mapped by block.
Fixes: 63b2bc619565 ("powerpc/mm/32s: Use BATs for STRICT_KERNEL_RWX")
Cc: stable@vger.kernel.org
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/mm/pgtable_32.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
When CONFIG_PTE_64BIT is set, pte_update() operates on
'unsigned long long'
When CONFIG_PTE_64BIT is not set, pte_update() operates on
'unsigned long'
In asm/page.h, we have pte_basic_t which is 'unsigned long long'
when CONFIG_PTE_64BIT is set and 'unsigned long' otherwise.
Refactor pte_update() using pte_basic_t.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/nohash/32/pgtable.h | 26 +++-----------------
1 file changed, 4 insertions(+), 22 deletions(-)
Commit 55c8fc3f4930 ("powerpc/8xx: reintroduce 16K pages with HW
assistance") redefined pte_t as a struct of 4 pte_basic_t, because
in 16K pages mode there are four identical entries in the page table.
But hugepd entries for 8M pages require only one entry of size
pte_basic_t. So there is no point in creating a cache for 4 entries
page tables.
Calculate PTE_T_ORDER using the size of pte_basic_t instead of pte_t.
Define specific huge_pte helpers (set_huge_pte_at(), huge_pte_clear(),
huge_ptep_set_wrprotect()) to write the pte in a single entry instead
of using set_pte_at() which writes 4 identical entries in 16k pages
mode. Also make sure that __ptep_set_access_flags() properly handle
the huge_pte case.
Define set_pte_filter() inline otherwise GCC doesn't inline it anymore
because it is now used twice, and that gives a pretty suboptimal code
because of pte_t being a struct of 4 entries.
Those functions are also used for 512k pages which only require one
entry as well allthough replicating it four times was harmless as 512k
pages entries are spread every 128 bytes in the table.
Signed-off-by: Christophe Leroy <redacted>
---
.../include/asm/nohash/32/hugetlb-8xx.h | 20 ++++++++++++++
arch/powerpc/include/asm/nohash/32/pgtable.h | 3 ++-
arch/powerpc/mm/hugetlbpage.c | 3 ++-
arch/powerpc/mm/pgtable.c | 26 ++++++++++++++++---
4 files changed, 46 insertions(+), 6 deletions(-)
Prepare ITLB handler to handle _PAGE_HUGE when CONFIG_HUGETLBFS
is enabled. This means that the L1 entry has to be kept in r11
until L2 entry is read, in order to insert _PAGE_HUGE into it.
Also move pgd_offset helpers before pte_update() as they
will be needed there in next patch.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/nohash/32/pgtable.h | 13 ++++++-------
arch/powerpc/kernel/head_8xx.S | 15 +++++++++------
2 files changed, 15 insertions(+), 13 deletions(-)
@@ -206,6 +206,12 @@ static inline void pmd_clear(pmd_t *pmdp)}+/* to find an entry in a kernel page-table-directory */+#define pgd_offset_k(address) pgd_offset(&init_mm, address)++/* to find an entry in a page-table-directory */+#define pgd_index(address) ((address) >> PGDIR_SHIFT)+#define pgd_offset(mm, address) ((mm)->pgd + pgd_index(address))/**PTEupdates.Thisfunctioniscalledwheneveranexisting
@@ -348,13 +354,6 @@ static inline int pte_young(pte_t pte)pfn_to_page((__pa(pmd_val(pmd))>>PAGE_SHIFT))#endif-/* to find an entry in a kernel page-table-directory */-#define pgd_offset_k(address) pgd_offset(&init_mm, address)--/* to find an entry in a page-table-directory */-#define pgd_index(address) ((address) >> PGDIR_SHIFT)-#define pgd_offset(mm, address) ((mm)->pgd + pgd_index(address))-/* Find an entry in the third-level page table.. */#define pte_index(address) \(((address)>>PAGE_SHIFT)&(PTRS_PER_PTE-1))
As the 8xx now manages 512k pages in standard page tables,
it doesn't need CONFIG_PPC_MM_SLICES anymore.
Don't select it anymore and remove all related code.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 64 --------------------
arch/powerpc/include/asm/nohash/32/slice.h | 20 ------
arch/powerpc/include/asm/slice.h | 2 -
arch/powerpc/platforms/Kconfig.cputype | 1 -
4 files changed, 87 deletions(-)
delete mode 100644 arch/powerpc/include/asm/nohash/32/slice.h
Similar to PPC64, accept to map RO data as ROX as a trade off between
between security and memory usage.
Having RO data executable is not a high risk as RO data can't be
modified to forge an exploit.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/Kconfig | 26 --------------------------
arch/powerpc/kernel/vmlinux.lds.S | 3 +--
2 files changed, 1 insertion(+), 28 deletions(-)
Map linear memory space with 512k and 8M pages whenever
possible.
Three mappings are performed:
- One for kernel text
- One for RO data
- One for the rest
Separating the mappings is done to be able to update the
protection later when using STRICT_KERNEL_RWX.
The ITLB miss handler now need to also handle huge TLBs
unless kernel text in pinned.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/head_8xx.S | 4 +--
arch/powerpc/mm/nohash/8xx.c | 50 +++++++++++++++++++++++++++++++++-
2 files changed, 51 insertions(+), 3 deletions(-)
DEBUG_PAGEALLOC only manages RW data.
Text and RO data can still be mapped with BATs.
In order to map with BATs, also enforce data alignment. Set
by default to 256M which is a good compromise for keeping
enough BATs for also KASAN and IMMR.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/Kconfig | 1 +
arch/powerpc/mm/book3s32/mmu.c | 6 ++++++
arch/powerpc/mm/init_32.c | 5 ++---
3 files changed, 9 insertions(+), 3 deletions(-)
@@ -170,6 +170,12 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)pr_debug("RAM mapped without BATs\n");returnbase;}+if(debug_pagealloc_enabled()){+if(base>=border)+returnbase;+if(top>=border)+top=border;+}if(!strict_kernel_rwx_enabled()||base>=border||top<=border)return__mmu_mapin_ram(base,top);
DEBUG_PAGEALLOC only manages RW data.
Text and RO data can still be mapped with hugepages and pinned TLB.
In order to map with hugepages, also enforce a 512kB data alignment
minimum. That's a trade-off between size of speed, taking into
account that DEBUG_PAGEALLOC is a debug option. Anyway the alignment
is still tunable.
We also allow tuning of alignment for book3s to limit the complexity
of the test in Kconfig that will anyway disappear in the following
patches once DEBUG_PAGEALLOC is handled together with BATs.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/Kconfig | 11 +++++++----
arch/powerpc/mm/init_32.c | 5 ++++-
arch/powerpc/mm/nohash/8xx.c | 11 ++++++++---
arch/powerpc/platforms/8xx/Kconfig | 2 +-
4 files changed, 20 insertions(+), 9 deletions(-)
@@ -149,7 +149,8 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top){unsignedlongetext8=ALIGN(__pa(_etext),SZ_8M);unsignedlongsinittext=__pa(_sinittext);-unsignedlongboundary=strict_kernel_rwx_enabled()?sinittext:etext8;+boolstrict_boundary=strict_kernel_rwx_enabled()||debug_pagealloc_enabled();+unsignedlongboundary=strict_boundary?sinittext:etext8;unsignedlongeinittext8=ALIGN(__pa(_einittext),SZ_8M);WARN_ON(top<einittext8);
@@ -160,8 +161,12 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)return0;mmu_mapin_ram_chunk(0,boundary,PAGE_KERNEL_TEXT,true);-mmu_mapin_ram_chunk(boundary,einittext8,PAGE_KERNEL_TEXT,true);-mmu_mapin_ram_chunk(einittext8,top,PAGE_KERNEL,true);+if(debug_pagealloc_enabled()){+top=boundary;+}else{+mmu_mapin_ram_chunk(boundary,einittext8,PAGE_KERNEL_TEXT,true);+mmu_mapin_ram_chunk(einittext8,top,PAGE_KERNEL,true);+}if(top>SZ_32M)memblock_set_current_limit(top);
Pinned TLB are 8M. Now that there is no strict boundary anymore
between text and RO data, it is possible to use 8M pinned executable
TLB that covers both text and RO data.
When PIN_TLB_DATA or PIN_TLB_TEXT is selected, enforce 8M RW data
alignment and allow STRICT_KERNEL_RWX.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/Kconfig | 8 +++++---
arch/powerpc/mm/nohash/8xx.c | 32 ++++++++++++++++++++++++++++++
arch/powerpc/platforms/8xx/Kconfig | 2 +-
3 files changed, 38 insertions(+), 4 deletions(-)
@@ -171,6 +171,20 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)returntop;}+staticvoidmmu_pin_text(unsignedlongboundary)+{+unsignedlongaddr=PAGE_OFFSET;+unsignedlongtwc=MD_SVALID|MD_PS8MEG;+unsignedlongrpn=__pa(addr)|0xf0|_PAGE_RO|+_PAGE_SPS|_PAGE_SH|_PAGE_PRESENT;+inti;++for(i=28;i<32&&__pa(addr)<boundary;i++,addr+=SZ_8M,rpn+=SZ_8M)+mpc8xx_update_tlb(0,i,addr|MI_EVALID,twc,rpn);+for(;i<32;i++)+mpc8xx_update_tlb(0,i,0,0,0);+}+voidmmu_mark_initmem_nx(void){unsignedlongetext8=ALIGN(__pa(_etext),SZ_8M);
Now that linear and IMMR dedicated TLB handling is gone, kernel
boundary address comparison is similar in ITLB miss handler and
in DTLB miss handler.
Create a macro named compare_to_kernel_boundary.
When TASK_SIZE is strictly below 0x80000000 and PAGE_OFFSET is
above 0x80000000, it is enough to compare to 0x8000000, and this
can be done with a single instruction.
Using not. instruction, we get to use 'blt' conditional branch as
when doing a regular comparison:
0x00000000 <= addr <= 0x7fffffff ==>
0xffffffff >= NOT(addr) >= 0x80000000
The above test corresponds to a 'blt'
Otherwise, do a regular comparison using two instructions.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/head_8xx.S | 22 ++++++++--------------
1 file changed, 8 insertions(+), 14 deletions(-)
Now that space have been freed next to the DTLB miss handler,
it's associated DTLB perf handling can be brought back in
the same place.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/head_8xx.S | 23 +++++++++++------------
1 file changed, 11 insertions(+), 12 deletions(-)
Add a function to early map kernel memory using huge pages.
For 512k pages, just use standard page table and map in using 512k
pages.
For 8M pages, create a hugepd table and populate the two PGD
entries with it.
This function can only be used to create page tables at startup. Once
the regular SLAB allocation functions replace memblock functions,
this function cannot allocate new pages anymore. However it can still
update existing mappings with new protections.
hugepd_none() macro is moved into asm/hugetlb.h to be usable outside
of mm/hugetlbpage.c
early_pte_alloc_kernel() is made visible.
_PAGE_HUGE flag is now displayed by ptdump.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/hugetlb.h | 2 +
.../include/asm/nohash/32/hugetlb-8xx.h | 5 ++
arch/powerpc/include/asm/pgtable.h | 2 +
arch/powerpc/mm/hugetlbpage.c | 2 -
arch/powerpc/mm/nohash/8xx.c | 52 +++++++++++++++++++
arch/powerpc/mm/pgtable_32.c | 2 +-
arch/powerpc/mm/ptdump/8xx.c | 5 ++
arch/powerpc/platforms/Kconfig.cputype | 1 +
8 files changed, 68 insertions(+), 3 deletions(-)
@@ -54,6 +56,56 @@ unsigned long p_block_mapped(phys_addr_t pa)return0;}+staticpte_t__init*early_hugepd_alloc_kernel(hugepd_t*pmdp,unsignedlongva)+{+if(hugepd_none(*pmdp)){+pte_t*ptep=memblock_alloc(sizeof(pte_basic_t),SZ_4K);++if(!ptep)+returnNULL;++hugepd_populate_kernel((hugepd_t*)pmdp,ptep,PAGE_SHIFT_8M);+hugepd_populate_kernel((hugepd_t*)pmdp+1,ptep,PAGE_SHIFT_8M);+}+returnhugepte_offset(*(hugepd_t*)pmdp,va,PGDIR_SHIFT);+}++staticint__ref__early_map_kernel_hugepage(unsignedlongva,phys_addr_tpa,+pgprot_tprot,intpsize,boolnew)+{+pmd_t*pmdp=pmd_ptr_k(va);+pte_t*ptep;++if(WARN_ON(psize!=MMU_PAGE_512K&&psize!=MMU_PAGE_8M))+return-EINVAL;++if(new){+if(WARN_ON(slab_is_available()))+return-EINVAL;++if(psize==MMU_PAGE_512K)+ptep=early_pte_alloc_kernel(pmdp,va);+else+ptep=early_hugepd_alloc_kernel((hugepd_t*)pmdp,va);+}else{+if(psize==MMU_PAGE_512K)+ptep=pte_offset_kernel(pmdp,va);+else+ptep=hugepte_offset(*(hugepd_t*)pmdp,va,PGDIR_SHIFT);+}++if(WARN_ON(!ptep))+return-ENOMEM;++/* The PTE should never be already present */+if(new&&WARN_ON(pte_present(*ptep)&&pgprot_val(prot)))+return-EINVAL;++set_huge_pte_at(&init_mm,va,ptep,pte_mkhuge(pfn_pte(pa>>PAGE_SHIFT,prot)));++return0;+}+/**MMU_init_hwdoesthechip-specificinitializationoftheMMUhardware.*/
The code to setup linear and IMMR mapping via huge TLB entries is
not called anymore. Remove it.
Also remove the handling of removed code exits in the perf driver.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 8 +-
arch/powerpc/kernel/head_8xx.S | 83 --------------------
arch/powerpc/perf/8xx-pmu.c | 10 ---
3 files changed, 1 insertion(+), 100 deletions(-)
Up to now, linear and IMMR mappings are managed via huge TLB entries
through specific code directly in TLB miss handlers. This implies
some patching of the TLB miss handlers at startup, and a lot of
dedicated code.
Remove all this specific dedicated code.
For now we are back to normal handling via standard 4k pages. In the
next patches, linear memory mapping and IMMR mapping will be managed
through huge pages.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/head_8xx.S | 29 +---------
arch/powerpc/mm/nohash/8xx.c | 103 +--------------------------------
2 files changed, 3 insertions(+), 129 deletions(-)
@@ -54,8 +54,6 @@ unsigned long p_block_mapped(phys_addr_t pa)return0;}-#define LARGE_PAGE_SIZE_8M (1<<23)-/**MMU_init_hwdoesthechip-specificinitializationoftheMMUhardware.*/
@@ -80,119 +78,20 @@ void __init mmu_mapin_immr(void)map_kernel_page(v+offset,p+offset,PAGE_KERNEL_NCG);}-staticvoidmmu_patch_cmp_limit(s32*site,unsignedlongmapped)-{-modify_instruction_site(site,0xffff,(unsignedlong)__va(mapped)>>16);-}--staticvoidmmu_patch_addis(s32*site,longsimm)-{-unsignedintinstr=*(unsignedint*)patch_site_addr(site);--instr&=0xffff0000;-instr|=((unsignedlong)simm)>>16;-patch_instruction_site(site,instr);-}--staticvoidmmu_mapin_ram_chunk(unsignedlongoffset,unsignedlongtop,pgprot_tprot)-{-unsignedlongs=offset;-unsignedlongv=PAGE_OFFSET+s;-phys_addr_tp=memstart_addr+s;--for(;s<top;s+=PAGE_SIZE){-map_kernel_page(v,p,prot);-v+=PAGE_SIZE;-p+=PAGE_SIZE;-}-}-unsignedlong__initmmu_mapin_ram(unsignedlongbase,unsignedlongtop){-unsignedlongmapped;-mmu_mapin_immr();-if(__map_without_ltlbs){-mapped=0;-if(!IS_ENABLED(CONFIG_PIN_TLB_IMMR))-patch_instruction_site(&patch__dtlbmiss_immr_jmp,PPC_INST_NOP);-if(!IS_ENABLED(CONFIG_PIN_TLB_TEXT))-mmu_patch_cmp_limit(&patch__itlbmiss_linmem_top,0);-}else{-unsignedlongeinittext8=ALIGN(__pa(_einittext),SZ_8M);--mapped=top&~(LARGE_PAGE_SIZE_8M-1);-if(!IS_ENABLED(CONFIG_PIN_TLB_TEXT))-mmu_patch_cmp_limit(&patch__itlbmiss_linmem_top,einittext8);--/*-*Populatepagetablesto:-*-havethemappearin/sys/kernel/debug/kernel_page_tables-*-allowtheBDItofindthepageswhentheyarenotPINNED-*/-mmu_mapin_ram_chunk(0,einittext8,PAGE_KERNEL_X);-mmu_mapin_ram_chunk(einittext8,mapped,PAGE_KERNEL);-}--mmu_patch_cmp_limit(&patch__dtlbmiss_linmem_top,mapped);-mmu_patch_cmp_limit(&patch__fixupdar_linmem_top,mapped);--/* If the size of RAM is not an exact power of two, we may not-*havecoveredRAMinitsentiretywith8MiB-*pages.Consequently,restrictthetopendofRAMcurrently-*allocablesothatcallstotheMEMBLOCKtoallocatePTEsfor"tail"-*coveragewithnormal-sizedpages(orotherreasons)donot-*attempttoallocateoutsidetheallowedrange.-*/-if(mapped)-memblock_set_current_limit(mapped);--block_mapped_ram=mapped;--returnmapped;+return0;}voidmmu_mark_initmem_nx(void){-if(IS_ENABLED(CONFIG_STRICT_KERNEL_RWX)&&CONFIG_ETEXT_SHIFT<23)-mmu_patch_addis(&patch__itlbmiss_linmem_top8,--((long)_etext&~(LARGE_PAGE_SIZE_8M-1)));-if(!IS_ENABLED(CONFIG_PIN_TLB_TEXT)){-unsignedlongeinittext8=ALIGN(__pa(_einittext),SZ_8M);-unsignedlongetext8=ALIGN(__pa(_etext),SZ_8M);-unsignedlongetext=__pa(_etext);--mmu_patch_cmp_limit(&patch__itlbmiss_linmem_top,__pa(_etext));--/* Update page tables for PTDUMP and BDI */-mmu_mapin_ram_chunk(0,einittext8,__pgprot(0));-if(IS_ENABLED(CONFIG_STRICT_KERNEL_RWX)){-mmu_mapin_ram_chunk(0,etext,PAGE_KERNEL_TEXT);-mmu_mapin_ram_chunk(etext,einittext8,PAGE_KERNEL);-}else{-mmu_mapin_ram_chunk(0,etext8,PAGE_KERNEL_TEXT);-mmu_mapin_ram_chunk(etext8,einittext8,PAGE_KERNEL);-}-}}#ifdef CONFIG_STRICT_KERNEL_RWXvoidmmu_mark_rodata_ro(void){-unsignedlongsinittext=__pa(_sinittext);-unsignedlongetext=__pa(_etext);--if(CONFIG_DATA_SHIFT<23)-mmu_patch_addis(&patch__dtlbmiss_romem_top8,--__pa(((unsignedlong)_sinittext)&-~(LARGE_PAGE_SIZE_8M-1)));-mmu_patch_addis(&patch__dtlbmiss_romem_top,-__pa(_sinittext));--/* Update page tables for PTDUMP and BDI */-mmu_mapin_ram_chunk(0,sinittext,__pgprot(0));-mmu_mapin_ram_chunk(0,etext,PAGE_KERNEL_ROX);-mmu_mapin_ram_chunk(etext,sinittext,PAGE_KERNEL_RO);}#endif
CONFIG_8xx_COPYBACK was there to help disabling copyback cache mode
for debuging hardware. But nobody will design new boards with 8xx now.
All 8xx platforms select it, so make it the default and remove
the option.
Also remove the Mx_RESETVAL values which are pretty useless and hide
the real value while reading code.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/configs/adder875_defconfig | 1 -
arch/powerpc/configs/ep88xc_defconfig | 1 -
arch/powerpc/configs/mpc866_ads_defconfig | 1 -
arch/powerpc/configs/mpc885_ads_defconfig | 1 -
arch/powerpc/configs/tqm8xx_defconfig | 1 -
arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 2 --
arch/powerpc/kernel/head_8xx.S | 15 +--------------
arch/powerpc/platforms/8xx/Kconfig | 9 ---------
8 files changed, 1 insertion(+), 30 deletions(-)
@@ -10,7 +10,6 @@ CONFIG_EXPERT=y # CONFIG_BLK_DEV_BSG is not set CONFIG_PARTITION_ADVANCED=y CONFIG_PPC_ADDER875=y-CONFIG_8xx_COPYBACK=y CONFIG_GEN_RTC=y CONFIG_HZ_1000=y # CONFIG_SECCOMP is not set
@@ -12,7 +12,6 @@ CONFIG_EXPERT=y # CONFIG_BLK_DEV_BSG is not set CONFIG_PARTITION_ADVANCED=y CONFIG_PPC_EP88XC=y-CONFIG_8xx_COPYBACK=y CONFIG_GEN_RTC=y CONFIG_HZ_100=y # CONFIG_SECCOMP is not set
@@ -12,7 +12,6 @@ CONFIG_EXPERT=y # CONFIG_BLK_DEV_BSG is not set CONFIG_PARTITION_ADVANCED=y CONFIG_MPC86XADS=y-CONFIG_8xx_COPYBACK=y CONFIG_GEN_RTC=y CONFIG_HZ_1000=y CONFIG_MATH_EMULATION=y
@@ -11,7 +11,6 @@ CONFIG_EXPERT=y # CONFIG_VM_EVENT_COUNTERS is not set # CONFIG_BLK_DEV_BSG is not set CONFIG_PARTITION_ADVANCED=y-CONFIG_8xx_COPYBACK=y CONFIG_GEN_RTC=y CONFIG_HZ_100=y # CONFIG_SECCOMP is not set
@@ -15,7 +15,6 @@ CONFIG_MODULE_SRCVERSION_ALL=y # CONFIG_BLK_DEV_BSG is not set CONFIG_PARTITION_ADVANCED=y CONFIG_TQM8XX=y-CONFIG_8xx_COPYBACK=y # CONFIG_8xx_CPU15 is not set CONFIG_GEN_RTC=y CONFIG_HZ_100=y
@@ -19,7 +19,6 @@#define MI_RSV4I 0x08000000 /* Reserve 4 TLB entries */#define MI_PPCS 0x02000000 /* Use MI_RPN prob/priv state */#define MI_IDXMASK 0x00001f00 /* TLB index to be loaded */-#define MI_RESETVAL 0x00000000 /* Value of register at reset *//* These are the Ks and Kp from the PowerPC books. For proper operation,*Ks=0,Kp=1.
@@ -95,7 +94,6 @@#define MD_TWAM 0x04000000 /* Use 4K page hardware assist */#define MD_PPCS 0x02000000 /* Use MI_RPN prob/priv state */#define MD_IDXMASK 0x00001f00 /* TLB index to be loaded */-#define MD_RESETVAL 0x04000000 /* Value of register at reset */#define SPRN_M_CASID 793 /* Address space ID (context) to match */#define MC_ASIDMASK 0x0000000f /* Bits used for ASID value */
@@ -98,15 +98,6 @@ menu "MPC8xx CPM Options"# 8xx specific questions.comment"Generic MPC8xx Options"-config8xx_COPYBACK-bool"Copy-Back Data Cache (else Writethrough)"-help-SayingYherewillcausethecacheonanMPC8xxprocessortobeused-inCopy-Backmode.IfyousayNhere,itisusedinWritethrough-mode.--Ifindoubt,sayYhere.-config8xx_GPIObool"GPIO API Support"selectGPIOLIB
Only early debug requires IMMR to be mapped early.
No need to set it up and pin it in assembly. Map it
through page tables at udbg init when necessary.
If CONFIG_PIN_TLB_IMMR is selected, pin it once we
don't need the 32 Mb pinned RAM anymore.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/head_8xx.S | 36 ++++++++++++------------------
arch/powerpc/mm/mmu_decl.h | 4 ++++
arch/powerpc/mm/nohash/8xx.c | 15 +++++++++----
arch/powerpc/platforms/8xx/Kconfig | 2 +-
arch/powerpc/sysdev/cpm_common.c | 2 ++
5 files changed, 32 insertions(+), 27 deletions(-)
@@ -121,9 +128,10 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top){unsignedlongmapped;+mmu_mapin_immr();+if(__map_without_ltlbs){mapped=0;-mmu_mapin_immr();if(!IS_ENABLED(CONFIG_PIN_TLB_IMMR))patch_instruction_site(&patch__dtlbmiss_immr_jmp,PPC_INST_NOP);if(!IS_ENABLED(CONFIG_PIN_TLB_TEXT))
@@ -142,7 +150,6 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)*/mmu_mapin_ram_chunk(0,einittext8,PAGE_KERNEL_X);mmu_mapin_ram_chunk(einittext8,mapped,PAGE_KERNEL);-mmu_mapin_immr();}mmu_patch_cmp_limit(&patch__dtlbmiss_linmem_top,mapped);
At startup, map 32 Mbytes of memory through 4 pages of 8M,
and PIN them inconditionnaly. They need to be pinned because
KASAN is using page tables early and the TLBs might be
dynamically replaced otherwise.
Remove RSV4I flag after installing mappings unless
CONFIG_PIN_TLB_XXXX is selected.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/head_8xx.S | 31 +++++++++++++++++--------------
arch/powerpc/mm/nohash/8xx.c | 19 +------------------
2 files changed, 18 insertions(+), 32 deletions(-)
@@ -61,23 +61,6 @@ unsigned long p_block_mapped(phys_addr_t pa)*/void__initMMU_init_hw(void){-/* PIN up to the 3 first 8Mb after IMMR in DTLB table */-if(IS_ENABLED(CONFIG_PIN_TLB_DATA)){-unsignedlongctr=mfspr(SPRN_MD_CTR)&0xfe000000;-unsignedlongflags=0xf0|MD_SPS16K|_PAGE_SH|_PAGE_DIRTY;-inti=28;-unsignedlongaddr=0;-unsignedlongmem=total_lowmem;--for(;i<32&&mem>=LARGE_PAGE_SIZE_8M;i++){-mtspr(SPRN_MD_CTR,ctr|(i<<8));-mtspr(SPRN_MD_EPN,(unsignedlong)__va(addr)|MD_EVALID);-mtspr(SPRN_MD_TWC,MD_PS8MEG|MD_SVALID);-mtspr(SPRN_MD_RPN,addr|flags|_PAGE_PRESENT);-addr+=LARGE_PAGE_SIZE_8M;-mem-=LARGE_PAGE_SIZE_8M;-}-}}staticboolimmr_is_mapped__initdata;
@@ -222,7 +205,7 @@ void __init setup_initial_memory_limit(phys_addr_t first_memblock_base,BUG_ON(first_memblock_base!=0);/* 8xx can only access 32MB at the moment */-memblock_set_current_limit(min_t(u64,first_memblock_size,0x02000000));+memblock_set_current_limit(min_t(u64,first_memblock_size,SZ_32M));}/*
Pinned TLBs are not easy to modify when the MMU is enabled.
Create a small function to update a pinned TLB entry with MMU off.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 3 ++
arch/powerpc/kernel/head_8xx.S | 44 ++++++++++++++++++++
2 files changed, 47 insertions(+)
PPC_PIN_TLB options are dedicated to the 8xx, move them into
the 8xx Kconfig.
While we are at it, add some text to explain what it does.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/Kconfig | 20 ---------------
arch/powerpc/platforms/8xx/Kconfig | 41 ++++++++++++++++++++++++++++++
2 files changed, 41 insertions(+), 20 deletions(-)
PPC64 takes 3 additional parameters compared to PPC32:
- mm
- address
- huge
These 3 parameters will be needed in order to perform different
action depending on the page size on the 8xx.
Make pte_update() prototype identical for PPC32 and PPC64.
This allows dropping an #ifdef in huge_ptep_get_and_clear().
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/book3s/32/pgtable.h | 15 ++++++++-------
arch/powerpc/include/asm/hugetlb.h | 4 ----
arch/powerpc/include/asm/nohash/32/pgtable.h | 13 +++++++------
3 files changed, 15 insertions(+), 17 deletions(-)
@@ -522,7 +523,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,*ptep=__pte((pte_val(*ptep)&_PAGE_HASHPTE)|(pte_val(pte)&~_PAGE_HASHPTE));else-pte_update(ptep,~_PAGE_HASHPTE,pte_val(pte));+pte_update(mm,addr,ptep,~_PAGE_HASHPTE,pte_val(pte),0);#elif defined(CONFIG_PTE_64BIT)/* Second case is 32-bit with 64-bit PTE. In this case, we
When CONFIG_PTE_64BIT is set, pte_update() operates on
'unsigned long long'
When CONFIG_PTE_64BIT is not set, pte_update() operates on
'unsigned long'
In asm/page.h, we have pte_basic_t which is 'unsigned long long'
when CONFIG_PTE_64BIT is set and 'unsigned long' otherwise.
Refactor pte_update() using pte_basic_t.
While we are at it, drop the comment on 44x which is not applicable
to book3s version of pte_update().
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/book3s/32/pgtable.h | 58 +++++++-------------
1 file changed, 20 insertions(+), 38 deletions(-)
512k pages are now standard pages, so only 8M pages
are hugepte.
No more handling of normal page tables through hugepd allocation
and freeing, and hugepte helpers can also be simplified.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h | 7 +++----
arch/powerpc/mm/hugetlbpage.c | 16 +++-------------
2 files changed, 6 insertions(+), 17 deletions(-)
At the time being, 512k huge pages are handled through hugepd page
tables. The PMD entry is flagged as a hugepd pointer and it
means that only 512k hugepages can be managed in that 4M block.
However, the hugepd table has the same size as a normal page
table, and 512k entries can therefore be nested with normal pages.
On the 8xx, TLB loading is performed by software and allthough the
page tables are organised to match the L1 and L2 level defined by
the HW, all TLB entries have both L1 and L2 independent entries.
It means that even if two TLB entries are associated with the same
PMD entry, they can be loaded with different values in L1 part.
The L1 entry contains the page size (PS field):
- 00 for 4k and 16 pages
- 01 for 512k pages
- 11 for 8M pages
By adding a flag for hugepages in the PTE (_PAGE_HUGE) and copying it
into the lower bit of PS, we can then manage 512k pages with normal
page tables:
- PMD entry has PS=11 for 8M pages
- PMD entry has PS=00 for other pages.
As a PMD entry covers 4M areas, a PMD will either point to a hugepd
table having a single entry to an 8M page, or the PMD will point to
a standard page table which will have either entries to 4k or 16k or
512k pages. For 512k pages, as the L1 entry will not know it is a
512k page before the PTE is read, there will be 128 entries in the
PTE as if it was 4k pages. But when loading the TLB, it will be
flagged as a 512k page.
Note that we can't use pmd_ptr() in asm/nohash/32/pgtable.h because
it is not defined yet.
In ITLB miss, we keep the possibility to opt it out as when kernel
text is pinned and no user hugepages are used, we can save several
instruction by not using r11.
In DTLB miss, that's just one instruction so it's not worth bothering
with it.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/nohash/32/pgtable.h | 10 ++++++---
arch/powerpc/include/asm/nohash/32/pte-8xx.h | 4 +++-
arch/powerpc/include/asm/nohash/pgtable.h | 2 +-
arch/powerpc/kernel/head_8xx.S | 12 +++++------
arch/powerpc/mm/hugetlbpage.c | 22 +++++++++++++++++---
arch/powerpc/mm/pgtable.c | 10 ++++++++-
6 files changed, 44 insertions(+), 16 deletions(-)
@@ -46,6 +46,8 @@#define _PAGE_NA 0x0200 /* Supervisor NA, User no access */#define _PAGE_RO 0x0600 /* Supervisor RO, User no access */+#define _PAGE_HUGE 0x0800 /* Copied to L1 PS bit 29 */+/* cache related flags non existing on 8xx */#define _PAGE_COHERENT 0#define _PAGE_WRITETHRU 0
@@ -267,7 +267,7 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,staticinlineinthugepd_ok(hugepd_thpd){#ifdef CONFIG_PPC_8xx-return((hpd_val(hpd)&0x4)!=0);+return((hpd_val(hpd)&_PMD_PAGE_MASK)==_PMD_PAGE_8M);#else/* We clear the top bit to indicate hugepd */return(hpd_val(hpd)&&(hpd_val(hpd)&PD_HUGE)==0);
pte_update() is a bit special for the 8xx. At the time
being, that's an #ifdef inside the nohash/32 pte_update().
As we are going to make it even more special in the coming
patches, create a dedicated version for pte_update() for 8xx.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/nohash/32/pgtable.h | 29 +++++++++++++++++---
1 file changed, 25 insertions(+), 4 deletions(-)
Allocate static page tables for the fixmap area. This allows
setting mappings through page tables before memblock is ready.
That's needed to use early_ioremap() early and to use standard
page mappings with fixmap.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/fixmap.h | 4 ++++
arch/powerpc/kernel/setup_32.c | 2 +-
arch/powerpc/mm/pgtable_32.c | 16 ++++++++++++++++
3 files changed, 21 insertions(+), 1 deletion(-)
@@ -81,7 +81,7 @@ notrace void __init machine_init(u64 dt_ptr)/* Configure static keys first, now that we're relocated. */setup_feature_keys();-early_ioremap_setup();+early_ioremap_init();/* Enable early debugging if any specified (see udbg.h) */udbg_early_init();
Mapping RO data as ROX is not an issue since that data
cannot be modified to introduce an exploit.
PPC64 accepts to have RO data mapped ROX, as a trade off
between kernel size and strictness of protection.
On PPC32, kernel size is even more critical as amount of
memory is usually small.
Depending on the number of available IBATs, the last IBATs
might overflow the end of text. Only warn if it crosses
the end of RO data.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/mm/book3s32/mmu.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
The 8xx is about to map kernel linear space and IMMR using huge
pages.
In order to display those pages properly, ptdump needs to handle
hugepd tables at PGD level.
For the time being do it only at PGD level. Further patches may
add handling of hugepd tables at lower level for other platforms
when needed in the future.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/mm/ptdump/ptdump.c | 29 ++++++++++++++++++++++++++---
1 file changed, 26 insertions(+), 3 deletions(-)
On PPC32, __ptep_test_and_clear_young() takes the mm->context.id
In preparation of standardising pte_update() params between PPC32 and
PPC64, __ptep_test_and_clear_young() need mm instead of mm->context.id
Replace context param by mm.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/book3s/32/pgtable.h | 7 ++++---
arch/powerpc/include/asm/nohash/32/pgtable.h | 5 +++--
2 files changed, 7 insertions(+), 5 deletions(-)
Only 40x still uses PTE_ATOMIC_UPDATES.
40x cannot not select CONFIG_PTE64_BIT.
Drop handling of PTE_ATOMIC_UPDATES:
- In nohash/64
- In nohash/32 for CONFIG_PTE_64BIT
Keep PTE_ATOMIC_UPDATES only for nohash/32 for !CONFIG_PTE_64BIT
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/nohash/32/pgtable.h | 17 ------------
arch/powerpc/include/asm/nohash/64/pgtable.h | 28 +-------------------
2 files changed, 1 insertion(+), 44 deletions(-)
In order to properly display information regardless of the page size,
it is necessary to take into account real page size.
Signed-off-by: Christophe Leroy <redacted>
Fixes: cabe8138b23c ("powerpc: dump as a single line areas mapping a single physical page.")
Cc: stable@vger.kernel.org
---
arch/powerpc/mm/ptdump/ptdump.c | 22 +++++++++++++---------
1 file changed, 13 insertions(+), 9 deletions(-)
@@ -195,10 +196,11 @@ static void note_prot_wx(struct pg_state *st, unsigned long addr)}staticvoidnote_page(structpg_state*st,unsignedlongaddr,-unsignedintlevel,u64val)+unsignedintlevel,u64val,intshift){u64flag=val&pg_level[level].mask;u64pa=val&PTE_RPN_MASK;+unsignedlongpage_size=1<<shift;/* At first no level is set */if(!st->level){
In order to have all flags fit on a 80 chars wide screen,
reduce the flags to 1 char (2 where ambiguous).
No cache is 'i'
User is 'ur' (Supervisor would be sr)
Shared (for 8xx) becomes 'sh' (it was 'user' when not shared but
that was ambiguous because that's not entirely right)
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/mm/ptdump/8xx.c | 33 ++++++++++++++++---------------
arch/powerpc/mm/ptdump/shared.c | 35 +++++++++++++++++----------------
2 files changed, 35 insertions(+), 33 deletions(-)
Commit 45ff3c559585 ("powerpc/kasan: Fix parallel loading of
modules.") added spinlocks to manage parallele module loading.
Since then commit 47febbeeec44 ("powerpc/32: Force KASAN_VMALLOC for
modules") converted the module loading to KASAN_VMALLOC.
The spinlocking has then become unneeded and can be removed to
simplify kasan_init_shadow_page_tables()
Also remove inclusion of linux/moduleloader.h and linux/vmalloc.h
which are not needed anymore since the removal of modules management.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/mm/kasan/kasan_init_32.c | 19 ++++---------------
1 file changed, 4 insertions(+), 15 deletions(-)
@@ -34,31 +32,22 @@ static int __init kasan_init_shadow_page_tables(unsigned long k_start, unsigned{pmd_t*pmd;unsignedlongk_cur,k_next;-pte_t*new=NULL;pmd=pmd_ptr_k(k_start);for(k_cur=k_start;k_cur!=k_end;k_cur=k_next,pmd++){+pte_t*new;+k_next=pgd_addr_end(k_cur,k_end);if((void*)pmd_page_vaddr(*pmd)!=kasan_early_shadow_pte)continue;-if(!new)-new=memblock_alloc(PTE_FRAG_SIZE,PTE_FRAG_SIZE);+new=memblock_alloc(PTE_FRAG_SIZE,PTE_FRAG_SIZE);if(!new)return-ENOMEM;kasan_populate_pte(new,PAGE_KERNEL);--smp_wmb();/* See comment in __pte_alloc */--spin_lock(&init_mm.page_table_lock);-/* Has another populated it ? */-if(likely((void*)pmd_page_vaddr(*pmd)==kasan_early_shadow_pte)){-pmd_populate_kernel(&init_mm,pmd,new);-new=NULL;-}-spin_unlock(&init_mm.page_table_lock);+pmd_populate_kernel(&init_mm,pmd,new);}return0;}
In case (k_start & PAGE_MASK) doesn't equal (kstart), 'va' will never be
NULL allthough 'block' is NULL
Check the return of memblock_alloc() directly instead of
the resulting address in the loop.
Fixes: 509cd3f2b473 ("powerpc/32: Simplify KASAN init")
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/mm/kasan/kasan_init_32.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
From: kbuild test robot <hidden> Date: 2020-03-17 01:40:33
Hi Christophe,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on next-20200316]
[cannot apply to powerpc/next v5.6-rc6 v5.6-rc5 v5.6-rc4 v5.6-rc6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Christophe-Leroy/Use-hugepages-to-map-kernel-mem-on-8xx/20200317-065610
base: 8548fd2f20ed19b0e8c0585b71fdfde1ae00ae3c
config: powerpc-tqm8xx_defconfig (attached as .config)
compiler: powerpc-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=powerpc
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <redacted>
All errors (new ones prefixed by >>):
In file included from arch/powerpc/mm/fault.c:33:
include/linux/hugetlb.h: In function 'hstate_inode':
quoted
include/linux/hugetlb.h:522:9: error: implicit declaration of function 'HUGETLBFS_SB'; did you mean 'HUGETLBFS_MAGIC'? [-Werror=implicit-function-declaration]
include/linux/hugetlb.h:522:30: error: invalid type argument of '->' (have 'int')
522 | return HUGETLBFS_SB(i->i_sb)->hstate;
| ^~
cc1: all warnings being treated as errors
--
In file included from arch/powerpc/mm/mem.c:30:
include/linux/hugetlb.h: In function 'hstate_inode':
quoted
include/linux/hugetlb.h:522:9: error: implicit declaration of function 'HUGETLBFS_SB' [-Werror=implicit-function-declaration]
include/linux/hugetlb.h:522:30: error: invalid type argument of '->' (have 'int')
522 | return HUGETLBFS_SB(i->i_sb)->hstate;
| ^~
cc1: all warnings being treated as errors
--
In file included from arch/powerpc/mm/nohash/8xx.c:12:
include/linux/hugetlb.h: In function 'hstate_inode':
quoted
include/linux/hugetlb.h:522:9: error: implicit declaration of function 'HUGETLBFS_SB' [-Werror=implicit-function-declaration]
include/linux/hugetlb.h:522:30: error: invalid type argument of '->' (have 'int')
522 | return HUGETLBFS_SB(i->i_sb)->hstate;
| ^~
At top level:
arch/powerpc/mm/nohash/8xx.c:73:18: error: '__early_map_kernel_hugepage' defined but not used [-Werror=unused-function]
73 | static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
--
In file included from arch/powerpc//mm/nohash/8xx.c:12:
include/linux/hugetlb.h: In function 'hstate_inode':
quoted
include/linux/hugetlb.h:522:9: error: implicit declaration of function 'HUGETLBFS_SB' [-Werror=implicit-function-declaration]
include/linux/hugetlb.h:522:30: error: invalid type argument of '->' (have 'int')
522 | return HUGETLBFS_SB(i->i_sb)->hstate;
| ^~
At top level:
arch/powerpc//mm/nohash/8xx.c:73:18: error: '__early_map_kernel_hugepage' defined but not used [-Werror=unused-function]
73 | static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
--
In file included from include/linux/migrate.h:8,
from kernel///sched/sched.h:53,
from kernel///sched/loadavg.c:9:
include/linux/hugetlb.h: In function 'hstate_inode':
quoted
include/linux/hugetlb.h:522:9: error: implicit declaration of function 'HUGETLBFS_SB'; did you mean 'HUGETLBFS_MAGIC'? [-Werror=implicit-function-declaration]
From: kbuild test robot <hidden> Date: 2020-03-17 03:25:17
Hi Christophe,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on next-20200316]
[cannot apply to powerpc/next v5.6-rc6 v5.6-rc5 v5.6-rc4 v5.6-rc6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Christophe-Leroy/Use-hugepages-to-map-kernel-mem-on-8xx/20200317-065610
base: 8548fd2f20ed19b0e8c0585b71fdfde1ae00ae3c
config: powerpc-rhel-kconfig (attached as .config)
compiler: powerpc64le-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=powerpc
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <redacted>
All warnings (new ones prefixed by >>):
In file included from arch/powerpc/mm/ptdump/book3s64.c:10:
quoted
arch/powerpc/mm/ptdump/ptdump.h:21:26: warning: 'struct seq_file' declared inside parameter list will not be visible outside of this definition or declaration
21 | void pt_dump_size(struct seq_file *m, unsigned long delta);
| ^~~~~~~~
vim +21 arch/powerpc/mm/ptdump/ptdump.h
20
> 21 void pt_dump_size(struct seq_file *m, unsigned long delta);
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
Implement a kasan_init_region() dedicated to book3s/32 that
allocates KASAN regions using BATs.
Signed-off-by: Christophe Leroy <redacted>
Note that the sparse warning on pmac32_defconfig is definitely a false
positive. See details patch 16/46 ("powerpc/mm: Allocate static page
tables for fixmap")
Christophe
Allocate static page tables for the fixmap area. This allows
setting mappings through page tables before memblock is ready.
That's needed to use early_ioremap() early and to use standard
page mappings with fixmap.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/fixmap.h | 4 ++++
arch/powerpc/kernel/setup_32.c | 2 +-
arch/powerpc/mm/pgtable_32.c | 16 ++++++++++++++++
3 files changed, 21 insertions(+), 1 deletion(-)
@@ -81,7 +81,7 @@ notrace void __init machine_init(u64 dt_ptr)/* Configure static keys first, now that we're relocated. */setup_feature_keys();-early_ioremap_setup();+early_ioremap_init();/* Enable early debugging if any specified (see udbg.h) */udbg_early_init();
Le 17/03/2020 à 02:39, kbuild test robot a écrit :
Hi Christophe,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on next-20200316]
[cannot apply to powerpc/next v5.6-rc6 v5.6-rc5 v5.6-rc4 v5.6-rc6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Christophe-Leroy/Use-hugepages-to-map-kernel-mem-on-8xx/20200317-065610
base: 8548fd2f20ed19b0e8c0585b71fdfde1ae00ae3c
config: powerpc-tqm8xx_defconfig (attached as .config)
compiler: powerpc-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=powerpc
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <redacted>
All errors (new ones prefixed by >>):
In file included from arch/powerpc/mm/fault.c:33:
include/linux/hugetlb.h: In function 'hstate_inode':
quoted
quoted
include/linux/hugetlb.h:522:9: error: implicit declaration of function 'HUGETLBFS_SB'; did you mean 'HUGETLBFS_MAGIC'? [-Werror=implicit-function-declaration]
include/linux/hugetlb.h:522:30: error: invalid type argument of '->' (have 'int')
522 | return HUGETLBFS_SB(i->i_sb)->hstate;
| ^~
At top level:
arch/powerpc//mm/nohash/8xx.c:73:18: error: '__early_map_kernel_hugepage' defined but not used [-Werror=unused-function]
73 | static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
This patch is a preparation patch. The function is not used yet, that's
normal. Ok, it breaks bisectability. Should it be squashed with the
first user of the function ?
Christophe
Sparse reports this as a variable size array. This is definitely not. Gcc
properly sees it is an 8k table (2 pages).
Yes, thing is that FIXMAP_PTE_SIZE is not that constant since it uses
__builtin_ffs() (via PTE_SHIFT / PTE_T_LOG2).
Nevertheless, since Sparse v0.6.1 (released in October) accepts these
in constant expressions, like GCC does.
-- Luc
Le 17/03/2020 à 02:39, kbuild test robot a écrit :
quoted
Hi Christophe,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on next-20200316]
[cannot apply to powerpc/next v5.6-rc6 v5.6-rc5 v5.6-rc4 v5.6-rc6]
[if your patch is applied to the wrong git tree, please drop us a note
to help
improve the system. BTW, we also suggest to use '--base' option to
specify the
base tree in git format-patch, please see
https://stackoverflow.com/a/37406982]
url:
https://github.com/0day-ci/linux/commits/Christophe-Leroy/Use-hugepages-to-map-kernel-mem-on-8xx/20200317-065610
base: 8548fd2f20ed19b0e8c0585b71fdfde1ae00ae3c
config: powerpc-tqm8xx_defconfig (attached as .config)
compiler: powerpc-linux-gcc (GCC) 9.2.0
reproduce:
wget
https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross
-O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=powerpc
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <redacted>
All errors (new ones prefixed by >>):
In file included from arch/powerpc/mm/fault.c:33:
include/linux/hugetlb.h: In function 'hstate_inode':
quoted
quoted
include/linux/hugetlb.h:522:9: error: implicit declaration of
function 'HUGETLBFS_SB'; did you mean 'HUGETLBFS_MAGIC'?
[-Werror=implicit-function-declaration]
include/linux/hugetlb.h:522:30: error: invalid type argument of '->'
(have 'int')
522 | return HUGETLBFS_SB(i->i_sb)->hstate;
| ^~
At top level:
arch/powerpc//mm/nohash/8xx.c:73:18: error:
'__early_map_kernel_hugepage' defined but not used
[-Werror=unused-function]
73 | static int __ref __early_map_kernel_hugepage(unsigned long
va, phys_addr_t pa,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
This patch is a preparation patch. The function is not used yet, that's
normal. Ok, it breaks bisectability. Should it be squashed with the
first user of the function ?
Christophe
Michael,
Le 16/03/2020 à 13:35, Christophe Leroy a écrit :
kasan_remap_early_shadow_ro() and kasan_unmap_early_shadow_vmalloc()
are both updating the early shadow mapping: the first one sets
the mapping read-only while the other clears the mapping.
Refactor and create kasan_update_early_region()
There is a trivial conflict with this patch on powerpc/next.
Do you plan to take this series for 5.7 ? I so, I can repost the series
now with the fix, or just this patch ?
Otherwise, what are your plans ? This series (Patches 18 and 19) will
conflict with the 40x removal series as both do things about that
PTE_ATOMIC_UPDATE stuff. Which series would go first ?
Thanks
Christophe