From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:44:54
Hi all,
This series reworks the arm64 early page table code. This avoids issues with
conflicting TLB entries, ensures we can always create page tables provided
there is available memory in the system (even if not covered by the early
linear map), and allows for fine-grained table creation early on.
With this series, we will create a new set of fine-grained page tables,
temporarily mapping the tables themselves using a new set of fixmap slots.
Subsequently we safely switch over to these tables using some idmapped code
that avoids the allocation of conflicting TLB entries.
This series is based on v4.5-rc1, and this version is tagged as
arm64-pagetable-rework-20160125 while the latest version should be in the
unstable branch arm64/pagetable-rework in my git repo [1], though I'm hoping
that this won't need another spin.
Since v1 [2] (tagged arm64-pagetable-rework-20151209):
* Drop patches taken into the arm64 tree.
* Rebase to arm64 for-next/core.
* Copy early KASAN tables.
* Fix KASAN pgd manipulation.
* Specialise allocators for page tables, in function and naming.
* Update comments.
Since v2 [3] (tagged arm64-pagetable-rework-20160104)
* Rebase to v4.5-rc1.
* Rename fixmap functions to p??_set_fixmap{,_offset}, p??_clear_fixmap.
* Restore a barrier lost to a rebase.
* Fix typos in commit messages.
* Collected Acks.
As there's at least one other series depending on this, I'd like to keep my
arm64/pagetable-rework branch stable.
Catalin, how do you want to take this? Are you happy to merge my
arm64/pagetable-rework branch, or would you prefer to pick the patches? Arnd
has acked the asm-generic patch [4] and is happy for it to go via the arm64
tree, and everything else is your domain.
Thanks,
Mark.
[1] git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git
[2] http://lists.infradead.org/pipermail/linux-arm-kernel/2015-December/392292.html
[3] http://lists.infradead.org/pipermail/linux-arm-kernel/2016-January/397095.html
[4] http://lists.infradead.org/pipermail/linux-arm-kernel/2016-January/400396.html
Mark Rutland (18):
asm-generic: make __set_fixmap_offset a static inline
arm64: mm: specialise pagetable allocators
arm64: mm: place empty_zero_page in bss
arm64: unify idmap removal
arm64: unmap idmap earlier
arm64: add function to install the idmap
arm64: mm: add code to safely replace TTBR1_EL1
arm64: kasan: avoid TLB conflicts
arm64: mm: move pte_* macros
arm64: mm: add functions to walk page tables by PA
arm64: mm: avoid redundant __pa(__va(x))
arm64: mm: add __{pud,pgd}_populate
arm64: mm: add functions to walk tables in fixmap
arm64: mm: use fixmap when creating page tables
arm64: mm: allocate pagetables anywhere
arm64: mm: allow passing a pgdir to alloc_init_*
arm64: ensure _stext and _etext are page-aligned
arm64: mm: create new fine-grained mappings at boot
arch/arm64/include/asm/fixmap.h | 10 ++
arch/arm64/include/asm/kasan.h | 3 +
arch/arm64/include/asm/mmu_context.h | 62 ++++++-
arch/arm64/include/asm/pgalloc.h | 26 ++-
arch/arm64/include/asm/pgtable.h | 87 +++++++---
arch/arm64/kernel/head.S | 1 +
arch/arm64/kernel/setup.c | 7 +
arch/arm64/kernel/smp.c | 4 +-
arch/arm64/kernel/suspend.c | 20 +--
arch/arm64/kernel/vmlinux.lds.S | 5 +-
arch/arm64/mm/kasan_init.c | 32 ++--
arch/arm64/mm/mmu.c | 313 ++++++++++++++++++-----------------
arch/arm64/mm/proc.S | 28 ++++
include/asm-generic/fixmap.h | 14 +-
14 files changed, 383 insertions(+), 229 deletions(-)
--
1.9.1
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:44:55
Currently __set_fixmap_offset is a macro function which has a local
variable called 'addr'. If a caller passes a 'phys' parameter which is
derived from a variable also called 'addr', the local variable will
shadow this, and the compiler will complain about the use of an
uninitialized variable.
It is likely that fixmap users may use the name 'addr' for variables
that may be directly passed to __set_fixmap_offset, or that may be
indirectly generated via other macros. Rather than placing the burden on
callers to avoid the name 'addr', this patch changes __set_fixmap_offset
into a static inline function, avoiding namespace collisions.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
include/asm-generic/fixmap.h | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:44:56
We pass a size parameter to early_alloc and late_alloc, but these are
only ever used to allocate single pages. In late_alloc we always
allocate a single page.
Both allocators provide us with zeroed pages (such that all entries are
invalid), but we have no barriers between allocating a page and adding
that page to existing (live) tables. A concurrent page table walk may
see stale data, leading to a number of issues.
This patch specialises the two allocators for page tables. The size
parameter is removed and the necessary dsb(ishst) is folded into each.
To make it clear that the functions are intended for use for page table
allocation, they are renamed to {early,late}_pgtable_alloc, with the
related function pointed renamed to pgtable_alloc.
As the dsb(ishst) is now in the allocator, the existing barrier for the
zero page is redundant and thus is removed. The previously missing
include of barrier.h is added.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/mm/mmu.c | 52 +++++++++++++++++++++++++++-------------------------
1 file changed, 27 insertions(+), 25 deletions(-)
@@ -62,15 +63,18 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,}EXPORT_SYMBOL(phys_mem_access_prot);-staticvoid__init*early_alloc(unsignedlongsz)+staticvoid__init*early_pgtable_alloc(void){phys_addr_tphys;void*ptr;-phys=memblock_alloc(sz,sz);+phys=memblock_alloc(PAGE_SIZE,PAGE_SIZE);BUG_ON(!phys);ptr=__va(phys);-memset(ptr,0,sz);+memset(ptr,0,PAGE_SIZE);++/* Ensure the zeroed page is visible to the page table walker */+dsb(ishst);returnptr;}
@@ -460,15 +465,12 @@ void __init paging_init(void)fixup_executable();/* allocate the zero page. */-zero_page=early_alloc(PAGE_SIZE);+zero_page=early_pgtable_alloc();bootmem_init();empty_zero_page=virt_to_page(zero_page);-/* Ensure the zero page is visible to the page table walker */-dsb(ishst);-/**TTBR0isonlyusedfortheidentitymappingatthisstage.Makeit*pointtozeropagetoavoidspeculativelyfetchingnewentries.
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:44:57
Currently the zero page is set up in paging_init, and thus we cannot use
the zero page earlier. We use the zero page as a reserved TTBR value
from which no TLB entries may be allocated (e.g. when uninstalling the
idmap). To enable such usage earlier (as may be required for invasive
changes to the kernel page tables), and to minimise the time that the
idmap is active, we need to be able to use the zero page before
paging_init.
This patch follows the example set by x86, by allocating the zero page
at compile time, in .bss. This means that the zero page itself is
available immediately upon entry to start_kernel (as we zero .bss before
this), and also means that the zero page takes up no space in the raw
Image binary. The associated struct page is allocated in bootmem_init,
and remains unavailable until this time.
Outside of arch code, the only users of empty_zero_page assume that the
empty_zero_page symbol refers to the zeroed memory itself, and that
ZERO_PAGE(x) must be used to acquire the associated struct page,
following the example of x86. This patch also brings arm64 inline with
these assumptions.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/include/asm/mmu_context.h | 2 +-
arch/arm64/include/asm/pgtable.h | 4 ++--
arch/arm64/kernel/head.S | 1 +
arch/arm64/mm/mmu.c | 9 +--------
4 files changed, 5 insertions(+), 11 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:44:58
We currently open-code the removal of the idmap and restoration of the
current task's MMU state in a few places.
Before introducing yet more copies of this sequence, unify these to call
a new helper, cpu_uninstall_idmap.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Lorenzo Pieralisi <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/include/asm/mmu_context.h | 25 +++++++++++++++++++++++++
arch/arm64/kernel/setup.c | 1 +
arch/arm64/kernel/smp.c | 4 +---
arch/arm64/kernel/suspend.c | 20 ++++----------------
arch/arm64/mm/mmu.c | 4 +---
5 files changed, 32 insertions(+), 22 deletions(-)
@@ -87,22 +86,11 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))ret=__cpu_suspend_enter(arg,fn);if(ret==0){/*-*WeareresumingfromresetwithTTBR0_EL1settothe-*idmaptoenabletheMMU;settheTTBR0tothereserved-*pagetablestopreventspeculativeTLBallocations,flush-*thelocaltlbandsetthedefaulttcr_el1.t0szsothat-*theTTBR0addressspaceset-upisproperlyrestored.-*Ifthecurrentactive_mm!=&init_mmweenteredcpu_suspend-*withmappingsinTTBR0thatmustberestored,soweswitch-*thembacktocompletetheaddressspaceconfiguration-*restorationbeforereturning.+*WeareresumingfromresetwiththeidmapactiveinTTBR0_EL1.+*WemustuninstalltheidmapandrestoretheexpectedMMU+*statebeforewecanpossiblyreturntouserspace.*/-cpu_set_reserved_ttbr0();-local_flush_tlb_all();-cpu_set_default_tcr_t0sz();--if(mm!=&init_mm)-cpu_switch_mm(mm->pgd,mm);+cpu_uninstall_idmap();/**Restoreper-cpuoffsetbeforeanykernel
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:44:59
During boot we leave the idmap in place until paging_init, as we
previously had to wait for the zero page to become allocated and
accessible.
Now that we have a statically-allocated zero page, we can uninstall the
idmap much earlier in the boot process, making it far easier to spot
accidental use of physical addresses. This also brings the cold boot
path in line with the secondary boot path.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/kernel/setup.c | 6 ++++++
arch/arm64/mm/mmu.c | 6 ------
2 files changed, 6 insertions(+), 6 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:00
In some cases (e.g. when making invasive changes to the kernel page
tables) we will need to execute code from the idmap.
Add a new helper which may be used to install the idmap, complementing
the existing cpu_uninstall_idmap.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/include/asm/mmu_context.h | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:01
If page tables are modified without suitable TLB maintenance, the ARM
architecture permits multiple TLB entries to be allocated for the same
VA. When this occurs, it is permitted that TLB conflict aborts are
raised in response to synchronous data/instruction accesses, and/or and
amalgamation of the TLB entries may be used as a result of a TLB lookup.
The presence of conflicting TLB entries may result in a variety of
behaviours detrimental to the system (e.g. erroneous physical addresses
may be used by I-cache fetches and/or page table walks). Some of these
cases may result in unexpected changes of hardware state, and/or result
in the (asynchronous) delivery of SError.
To avoid these issues, we must avoid situations where conflicting
entries may be allocated into TLBs. For user and module mappings we can
follow a strict break-before-make approach, but this cannot work for
modifications to the swapper page tables that cover the kernel text and
data.
Instead, this patch adds code which is intended to be executed from the
idmap, which can safely unmap the swapper page tables as it only
requires the idmap to be active. This enables us to uninstall the active
TTBR1_EL1 entry, invalidate TLBs, then install a new TTBR1_EL1 entry
without potentially unmapping code or data required for the sequence.
This avoids the risk of conflict, but requires that updates are staged
in a copy of the swapper page tables prior to being installed.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/include/asm/mmu_context.h | 19 +++++++++++++++++++
arch/arm64/mm/proc.S | 28 ++++++++++++++++++++++++++++
2 files changed, 47 insertions(+)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:02
The page table modification performed during the KASAN init risks the
allocation of conflicting TLB entries, as it swaps a set of valid global
entries for another without suitable TLB maintenance.
The presence of conflicting TLB entries can result in the delivery of
synchronous TLB conflict aborts, or may result in the use of erroneous
data being returned in response to a TLB lookup. This can affect
explicit data accesses from software as well as translations performed
asynchronously (e.g. as part of page table walks or speculative I-cache
fetches), and can therefore result in a wide variety of problems.
To avoid this, use cpu_replace_ttbr1 to swap the page tables. This
ensures that when the new tables are installed there are no stale
entries from the old tables which may conflict. As all updates are made
to the tables while they are not active, the updates themselves are
safe.
At the same time, add the missing barrier to ensure that the tmp_pg_dir
entries updated via memcpy are visible to the page table walkers at the
point the tmp_pg_dir is installed. All other page table updates made as
part of KASAN initialisation have the requisite barriers due to the use
of the standard page table accessors.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/mm/kasan_init.c | 17 ++++-------------
1 file changed, 4 insertions(+), 13 deletions(-)
@@ -156,8 +148,7 @@ void __init kasan_init(void)}memset(kasan_zero_page,0,PAGE_SIZE);-cpu_set_ttbr1(__pa(swapper_pg_dir));-flush_tlb_all();+cpu_replace_ttbr1(swapper_pg_dir);/* At this point kasan is fully initialized. Enable error messages */init_task.kasan_depth=0;
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:03
For pmd, pud, and pgd levels of table, functions including p?d_index and
p?d_offset are defined after the p?d_page_vaddr function for the
immediately higher level of table.
The pte functions however are defined much earlier, even though several
rely on the later definition of pmd_page_vaddr. While this isn't
currently a problem as these are macros, it prevents the logical
grouping of later C functions (which cannot rely on prototypes for
functions not yet defined).
Move these definitions after pmd_page_vaddr, for consistency with the
placement of these functions for other levels of table.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/include/asm/pgtable.h | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:04
To allow us to walk tables allocated into the fixmap, we need to acquire
the physical address of a page, rather than the virtual address in the
linear map.
This patch adds new p??_page_paddr and p??_offset_phys functions to
acquire the physical address of a next-level table, and changes
p??_offset* into macros which simply convert this to a linear map VA.
This renders p??_page_vaddr unused, and hence they are removed.
At the pgd level, a new pgd_offset_raw function is added to find the
relevant PGD entry given the base of a PGD and a virtual address.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/include/asm/pgtable.h | 39 +++++++++++++++++++++++----------------
1 file changed, 23 insertions(+), 16 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:05
When we "upgrade" to a section mapping, we free any table we made
redundant by giving it back to memblock. To get the PA, we acquire the
physical address and convert this to a VA, then subsequently convert
this back to a PA.
This works currently, but will not work if the tables are not accessed
via linear map VAs (e.g. is we use fixmap slots).
This patch uses {pmd,pud}_page_paddr to acquire the PA. This avoids the
__pa(__va()) round trip, saving some work and avoiding reliance on the
linear mapping.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/mm/mmu.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:06
We currently have __pmd_populate for creating a pmd table entry given
the physical address of a pte, but don't have equivalents for the pud or
pgd levels of table.
To enable us to manipulate tables which are mapped outside of the linear
mapping (where we have a PA, but not a linear map VA), it is useful to
have these functions.
This patch adds __{pud,pgd}_populate. As these should not be called when
the kernel uses folded {pmd,pud}s, in these cases they expand to
BUILD_BUG(). So long as the appropriate checks are made on the {pud,pgd}
entry prior to attempting population, these should be optimized out at
compile time.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/include/asm/pgalloc.h | 26 ++++++++++++++++++++++----
1 file changed, 22 insertions(+), 4 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:07
As a preparatory step to allow us to allocate early page tables from
unmapped memory using memblock_alloc, add new p??_{set,clear}_fixmap*
functions which can be used to walk page tables outside of the linear
mapping by using fixmap slots.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/include/asm/fixmap.h | 10 ++++++++++
arch/arm64/include/asm/pgtable.h | 26 ++++++++++++++++++++++++++
2 files changed, 36 insertions(+)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:08
As a preparatory step to allow us to allocate early page tables from
unmapped memory using memblock_alloc, modify the __create_mapping
callees to map and unmap the tables they modify using fixmap entries.
All but the top-level pgd initialisation is performed via the fixmap.
Subsequent patches will inject the pgd physical address, and migrate to
using the FIX_PGD slot.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/mm/mmu.c | 61 +++++++++++++++++++++++++++++++++++------------------
1 file changed, 41 insertions(+), 20 deletions(-)
@@ -63,19 +63,30 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,}EXPORT_SYMBOL(phys_mem_access_prot);-staticvoid__init*early_pgtable_alloc(void)+staticphys_addr_t__initearly_pgtable_alloc(void){phys_addr_tphys;void*ptr;phys=memblock_alloc(PAGE_SIZE,PAGE_SIZE);BUG_ON(!phys);-ptr=__va(phys);++/*+*TheFIX_{PGD,PUD,PMD}slotsmaybeinactiveuse,buttheFIX_PTE+*slotwillbefree,sowecan(ab)usetheFIX_PTEslottoinitialise+*anyleveloftable.+*/+ptr=pte_set_fixmap(phys);+memset(ptr,0,PAGE_SIZE);-/* Ensure the zeroed page is visible to the page table walker */-dsb(ishst);-returnptr;+/*+*Implicitbarriersalsoensurethezeroedpageisvisibletothepage+*tablewalker+*/+pte_clear_fixmap();++returnphys;}/*
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:09
Now that create_mapping uses fixmap slots to modify pte, pmd, and pud
entries, we can access page tables anywhere in physical memory,
regardless of the extent of the linear mapping.
Given that, we no longer need to limit memblock allocations during page
table creation, and can leave the limit as its default
MEMBLOCK_ALLOC_ANYWHERE.
We never add memory which will fall outside of the linear map range
given phys_offset and MAX_MEMBLOCK_ADDR are configured appropriately, so
any tables we create will fall in the linear map of the final tables.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/mm/mmu.c | 35 -----------------------------------
1 file changed, 35 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:10
To allow us to initialise pgdirs which are fixmapped, allow explicitly
passing a pgdir rather than an mm. A new __create_pgd_mapping function
is added for this, with existing __create_mapping callers migrated to
this.
The mm argument was previously only used at the top level. Now that it
is redundant at all levels, it is removed. To indicate its new found
similarity to alloc_init_{pud,pmd,pte}, __create_mapping is renamed to
init_pgd.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/mm/mmu.c | 33 +++++++++++++++++++--------------
1 file changed, 19 insertions(+), 14 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:11
Currently we have separate ALIGN_DEBUG_RO{,_MIN} directives to align
_etext and __init_begin. While we ensure that __init_begin is
page-aligned, we do not provide the same guarantee for _etext. This is
not problematic currently as the alignment of __init_begin is sufficient
to prevent issues when we modify permissions.
Subsequent patches will assume page alignment of segments of the kernel
we wish to map with different permissions. To ensure this, move _etext
after the ALIGN_DEBUG_RO_MIN for the init section. This renders the
prior ALIGN_DEBUG_RO irrelevant, and hence it is removed. Likewise,
upgrade to ALIGN_DEBUG_RO_MIN(PAGE_SIZE) for _stext.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/kernel/vmlinux.lds.S | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
From: Mark Rutland <mark.rutland@arm.com> Date: 2016-01-25 11:45:12
At boot we may change the granularity of the tables mapping the kernel
(by splitting or making sections). This may happen when we create the
linear mapping (in __map_memblock), or at any point we try to apply
fine-grained permissions to the kernel (e.g. fixup_executable,
mark_rodata_ro, fixup_init).
Changing the active page tables in this manner may result in multiple
entries for the same address being allocated into TLBs, risking problems
such as TLB conflict aborts or issues derived from the amalgamation of
TLB entries. Generally, a break-before-make (BBM) approach is necessary
to avoid conflicts, but we cannot do this for the kernel tables as it
risks unmapping text or data being used to do so.
Instead, we can create a new set of tables from scratch in the safety of
the existing mappings, and subsequently migrate over to these using the
new cpu_replace_ttbr1 helper, which avoids the two sets of tables being
active simultaneously.
To avoid issues when we later modify permissions of the page tables
(e.g. in fixup_init), we must create the page tables at a granularity
such that later modification does not result in splitting of tables.
This patch applies this strategy, creating a new set of fine-grained
page tables from scratch, and safely migrating to them. The existing
fixmap and kasan shadow page tables are reused in the new fine-grained
tables.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Tested-by: Ard Biesheuvel <redacted>
Reviewed-by: Ard Biesheuvel <redacted>
Cc: Jeremy Linton <redacted>
Cc: Laura Abbott <redacted>
Cc: Will Deacon <redacted>
---
arch/arm64/include/asm/kasan.h | 3 +
arch/arm64/mm/kasan_init.c | 15 ++++
arch/arm64/mm/mmu.c | 153 ++++++++++++++++++++++++-----------------
3 files changed, 109 insertions(+), 62 deletions(-)
@@ -344,49 +345,42 @@ static void create_mapping_late(phys_addr_t phys, unsigned long virt,late_pgtable_alloc);}-#ifdef CONFIG_DEBUG_RODATA-staticvoid__init__map_memblock(phys_addr_tstart,phys_addr_tend)+staticvoid__init__map_memblock(pgd_t*pgd,phys_addr_tstart,phys_addr_tend){++unsignedlongkernel_start=__pa(_stext);+unsignedlongkernel_end=__pa(_end);+/*-*Setuptheexecutableregionsusingtheexistingsectionmappings-*fornow.Thiswillgetmorefinegrainedlateronceallmemory-*ismapped+*Thekernelitselfismappedatpagegranularity.Mapallother+*memory,makingsurewedon'toverwritetheexistingkernelmappings.*/-unsignedlongkernel_x_start=round_down(__pa(_stext),SWAPPER_BLOCK_SIZE);-unsignedlongkernel_x_end=round_up(__pa(__init_end),SWAPPER_BLOCK_SIZE);--if(end<kernel_x_start){-create_mapping(start,__phys_to_virt(start),-end-start,PAGE_KERNEL);-}elseif(start>=kernel_x_end){-create_mapping(start,__phys_to_virt(start),-end-start,PAGE_KERNEL);-}else{-if(start<kernel_x_start)-create_mapping(start,__phys_to_virt(start),-kernel_x_start-start,-PAGE_KERNEL);-create_mapping(kernel_x_start,-__phys_to_virt(kernel_x_start),-kernel_x_end-kernel_x_start,-PAGE_KERNEL_EXEC);-if(kernel_x_end<end)-create_mapping(kernel_x_end,-__phys_to_virt(kernel_x_end),-end-kernel_x_end,-PAGE_KERNEL);++/* No overlap with the kernel. */+if(end<kernel_start||start>=kernel_end){+__create_pgd_mapping(pgd,start,__phys_to_virt(start),+end-start,PAGE_KERNEL,+early_pgtable_alloc);+return;}+/*+*Thisblockoverlapsthekernelmapping.Maptheportion(s)which+*don'toverlap.+*/+if(start<kernel_start)+__create_pgd_mapping(pgd,start,+__phys_to_virt(start),+kernel_start-start,PAGE_KERNEL,+early_pgtable_alloc);+if(kernel_end<end)+__create_pgd_mapping(pgd,kernel_end,+__phys_to_virt(kernel_end),+end-kernel_end,PAGE_KERNEL,+early_pgtable_alloc);}-#else-staticvoid__init__map_memblock(phys_addr_tstart,phys_addr_tend)-{-create_mapping(start,__phys_to_virt(start),end-start,-PAGE_KERNEL_EXEC);-}-#endif-staticvoid__initmap_mem(void)+staticvoid__initmap_mem(pgd_t*pgd){structmemblock_region*reg;
@@ -400,33 +394,10 @@ static void __init map_mem(void)if(memblock_is_nomap(reg))continue;-__map_memblock(start,end);+__map_memblock(pgd,start,end);}}-staticvoid__initfixup_executable(void)-{-#ifdef CONFIG_DEBUG_RODATA-/* now that we are actually fully mapped, make the start/end more fine grained */-if(!IS_ALIGNED((unsignedlong)_stext,SWAPPER_BLOCK_SIZE)){-unsignedlongaligned_start=round_down(__pa(_stext),-SWAPPER_BLOCK_SIZE);--create_mapping(aligned_start,__phys_to_virt(aligned_start),-__pa(_stext)-aligned_start,-PAGE_KERNEL);-}--if(!IS_ALIGNED((unsignedlong)__init_end,SWAPPER_BLOCK_SIZE)){-unsignedlongaligned_end=round_up(__pa(__init_end),-SWAPPER_BLOCK_SIZE);-create_mapping(__pa(__init_end),(unsignedlong)__init_end,-aligned_end-__pa(__init_end),-PAGE_KERNEL);-}-#endif-}-#ifdef CONFIG_DEBUG_RODATAvoidmark_rodata_ro(void){
From: Jeremy Linton <hidden> Date: 2016-01-27 15:43:19
On 01/25/2016 05:44 AM, Mark Rutland wrote:
Hi all,
This series reworks the arm64 early page table code. This avoids issues with
conflicting TLB entries, ensures we can always create page tables provided
there is available memory in the system (even if not covered by the early
linear map), and allows for fine-grained table creation early on.
I've tested this in a couple machines using ACPI based kernels,
including the xgene which initially had problems. I've tested it with
and without CONFIG_DEBUG_RODATA. Everything seems fine.
Thanks,
Tested-by: Jeremy Linton <redacted>
On Mon, Jan 25, 2016 at 11:44:54AM +0000, Mark Rutland wrote:
Catalin, how do you want to take this? Are you happy to merge my
arm64/pagetable-rework branch, or would you prefer to pick the patches? Arnd
has acked the asm-generic patch [4] and is happy for it to go via the arm64
tree, and everything else is your domain.
I picked them from the list and added Jeremy's Tested-by. I pushed the
branch here in case anyone wants to base their patches on top (I don't
plan to rebase it):
git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux for-next/pgtable
Thanks.
--
Catalin