Apologies for the wall of text. This is a followup to 'restrict virt_to_page
to linear region (instead of __pa)' [1], posted on the 24th of February.
This v2 series applies onto v4.6-rc1 with the series 'arm64: memstart_addr
alignment and vmemmap offset fixes' [2] applied on top.
Only minor changes since v1, primarily replacing 'x >> PAGE_SHIFT' instances
with PHYS_PFN, and rebasing onto the latest upstream.
When building the arm64 defconfig kernel, which has CONFIG_SPARSEMEM_VMEMMAP
enabled, the implementations of virt_to_page and its converse
[lowmem_]page_address resolve to
virt_to_page
6f8: b6300180 tbz x0, #38, 728 <bar+0x30>
6fc: 90000001 adrp x1, 0 <memstart_addr>
700: 92409400 and x0, x0, #0x3fffffffff
704: f9400021 ldr x1, [x1]
708: 8b000020 add x0, x1, x0
70c: 37000261 tbnz w1, #0, 758 <bar+0x60>
710: d34cfc00 lsr x0, x0, #12
714: d2dff7c2 mov x2, #0xffbe00000000
718: cb813000 sub x0, x0, x1, asr #12
71c: f2ffffe2 movk x2, #0xffff, lsl #48
720: 8b001840 add x0, x2, x0, lsl #6
724: d65f03c0 ret
728: 90000002 adrp x2, 0 <init_pgd>
72c: 90000001 adrp x1, 0 <memstart_addr>
730: f9400042 ldr x2, [x2]
734: f9400021 ldr x1, [x1]
738: cb020000 sub x0, x0, x2
73c: d2dff7c2 mov x2, #0xffbe00000000
740: d34cfc00 lsr x0, x0, #12
744: f2ffffe2 movk x2, #0xffff, lsl #48
748: cb813000 sub x0, x0, x1, asr #12
74c: 8b001840 add x0, x2, x0, lsl #6
750: d65f03c0 ret
754: d503201f nop
758: d4210000 brk #0x800
page_address:
6c0: 90000002 adrp x2, 0 <memstart_addr>
6c4: d2c00841 mov x1, #0x4200000000
6c8: f9400043 ldr x3, [x2]
6cc: 934cfc62 asr x2, x3, #12
6d0: 8b021800 add x0, x0, x2, lsl #6
6d4: 8b010001 add x1, x0, x1
6d8: 9346fc21 asr x1, x1, #6
6dc: d374cc21 lsl x1, x1, #12
6e0: 37000083 tbnz w3, #0, 6f0 <foo+0x30>
6e4: cb030020 sub x0, x1, x3
6e8: b25a6400 orr x0, x0, #0xffffffc000000000
6ec: d65f03c0 ret
6f0: d4210000 brk #0x800
Disappointingly, even though this translation is independent of the physical
start of RAM since commit dfd55ad85e ("arm64: vmemmap: use virtual projection
of linear region"), the expression is evaluated in a way that does not allow
the compiler to eliminate the read of memstart_addr, presumably since it is
unaware that its value is aligned to PAGE_SIZE, and that shifting it down and
up again by PAGE_SHIFT bits produces the exact same value.
So let's give the compiler a hand here. First of all, let's reimplement
virt_to_page() (patch #6) so that it explicitly translates without taking
the physical placement into account. This results in the virt_to_page()
translation to only work correctly for addresses above PAGE_OFFSET, but
this is a reasonable restriction to impose, even if it means a couple of
incorrect uses need to be fixed (patches #1 to #4). If we also, in patch #5,
move the vmemmap region right below the linear region (which guarantees that
the region is always aligned to a power-of-2 upper bound of its size, which
means we can treat VMEMMAP_START as a bitmask rather than an offset), we end
up with
virt_to_page
6d0: d34c9400 ubfx x0, x0, #12, #26
6d4: d2dff7c1 mov x1, #0xffbe00000000
6d8: f2ffffe1 movk x1, #0xffff, lsl #48
6dc: aa001820 orr x0, x1, x0, lsl #6
6e0: d65f03c0 ret
In the same way, we can get page_address to look like this
page_address:
6c0: d37a7c00 ubfiz x0, x0, #6, #32
6c4: b25a6400 orr x0, x0, #0xffffffc000000000
6c8: d65f03c0 ret
However, in this case, we need to slightly refactor the implementation of
lowmem_page_paddress(), since it performs an explicit page-to-pa-to-va
translation, rather than going through an opaque arch-defined definition
of page_to_virt. (patches #7 to #9)
[1] http://thread.gmane.org/gmane.linux.ports.arm.kernel/481327
[2] http://thread.gmane.org/gmane.linux.ports.arm.kernel/488876
Ard Biesheuvel (9):
arm64: vdso: avoid virt_to_page() translations on kernel symbols
arm64: mm: free __init memory via the linear mapping
arm64: mm: avoid virt_to_page() translation for the zero page
arm64: insn: avoid virt_to_page() translations on core kernel symbols
arm64: mm: move vmemmap region right below the linear region
arm64: mm: restrict virt_to_page() to the linear mapping
nios2: use correct void* return type for page_to_virt()
openrisc: drop wrongly typed definition of page_to_virt()
mm: replace open coded page to virt conversion with page_to_virt()
arch/arm64/include/asm/memory.h | 30 ++++++++++++++++++--
arch/arm64/include/asm/pgtable.h | 13 +++------
arch/arm64/kernel/insn.c | 2 +-
arch/arm64/kernel/vdso.c | 4 +--
arch/arm64/mm/dump.c | 16 +++++------
arch/arm64/mm/init.c | 17 +++++++----
arch/nios2/include/asm/io.h | 1 -
arch/nios2/include/asm/page.h | 2 +-
arch/nios2/include/asm/pgtable.h | 2 +-
arch/openrisc/include/asm/page.h | 2 --
include/linux/mm.h | 6 +++-
11 files changed, 62 insertions(+), 33 deletions(-)
--
2.5.0
The translation performed by virt_to_page() is only valid for linear
addresses, and kernel symbols are no longer in the linear mapping.
So perform the __pa() translation explicitly, which does the right
thing in either case, and only then translate to a struct page offset.
Signed-off-by: Ard Biesheuvel <redacted>
---
arch/arm64/kernel/vdso.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
@@ -131,11 +131,11 @@ static int __init vdso_init(void)return-ENOMEM;/* Grab the vDSO data page. */-vdso_pagelist[0]=virt_to_page(vdso_data);+vdso_pagelist[0]=pfn_to_page(PHYS_PFN(__pa(vdso_data)));/* Grab the vDSO code pages. */for(i=0;i<vdso_pages;i++)-vdso_pagelist[i+1]=virt_to_page(&vdso_start+i*PAGE_SIZE);+vdso_pagelist[i+1]=pfn_to_page(PHYS_PFN(__pa(&vdso_start))+i);/* Populate the special mapping structures */vdso_spec[0]=(structvm_special_mapping){
The implementation of free_initmem_default() expects __init_begin
and __init_end to be covered by the linear mapping, which is no
longer the case. So open code it instead, using addresses that are
explicitly translated from kernel virtual to linear virtual.
Signed-off-by: Ard Biesheuvel <redacted>
---
arch/arm64/mm/init.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
The zero page is statically allocated, so grab its struct page pointer
without using virt_to_page(), which will be restricted to the linear
mapping later.
Signed-off-by: Ard Biesheuvel <redacted>
---
arch/arm64/include/asm/pgtable.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Before restricting virt_to_page() to the linear mapping, ensure that
the text patching code does not use it to resolve references into the
core kernel text, which is mapped in the vmalloc area.
Signed-off-by: Ard Biesheuvel <redacted>
---
arch/arm64/kernel/insn.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
This moves the vmemmap region right below PAGE_OFFSET, aka the start
of the linear region, and redefines its size to be a power of two.
Due to the placement of PAGE_OFFSET in the middle of the address space,
whose size is a power of two as well, this guarantees that virt to
page conversions and vice versa can be implemented efficiently, by
masking and shifting rather than ordinary arithmetic.
Signed-off-by: Ard Biesheuvel <redacted>
---
arch/arm64/include/asm/memory.h | 18 +++++++++++++++++-
arch/arm64/include/asm/pgtable.h | 11 +++--------
arch/arm64/mm/dump.c | 16 ++++++++--------
arch/arm64/mm/init.c | 14 ++++++++++----
4 files changed, 38 insertions(+), 21 deletions(-)
Now that the vmemmap region has been redefined to cover the linear region
rather than the entire physical address space, we no longer need to
perform a virtual-to-physical translation in the implementaion of
virt_to_page(). This restricts virt_to_page() translations to the linear
region, so redefine virt_addr_valid() as well.
Signed-off-by: Ard Biesheuvel <redacted>
---
arch/arm64/include/asm/memory.h | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
To align with other architectures, the expression procuded by expanding
the macro page_to_virt() should be of type void*, since it returns a
virtual address. Fix that, and also fix up an instance where page_to_virt
was expected to return 'unsigned long', and drop another instance that was
entirely unused (page_to_bus)
Signed-off-by: Ard Biesheuvel <redacted>
---
arch/nios2/include/asm/io.h | 1 -
arch/nios2/include/asm/page.h | 2 +-
arch/nios2/include/asm/pgtable.h | 2 +-
3 files changed, 2 insertions(+), 3 deletions(-)
To align with generic code and other architectures that expect the macro
page_to_virt to produce an expression whose type is 'void*', drop the
arch specific definition, which is never referenced anyway.
Signed-off-by: Ard Biesheuvel <redacted>
---
arch/openrisc/include/asm/page.h | 2 --
1 file changed, 2 deletions(-)
The open coded conversion from struct page address to virtual address in
lowmem_page_address() involves an intermediate conversion step to pfn
number/physical address. Since the placement of the struct page array
relative to the linear mapping may be completely independent from the
placement of physical RAM (as is that case for arm64 after commit
dfd55ad85e 'arm64: vmemmap: use virtual projection of linear region'),
the conversion to physical address and back again should factor out of
the equation, but unfortunately, the shifting and pointer arithmetic
involved prevent this from happening, and the resulting calculation
essentially subtracts the address of the start of physical memory and
adds it back again, in a way that prevents the compiler from optimizing
it away.
Since the start of physical memory is not a build time constant on arm64,
the resulting conversion involves an unnecessary memory access, which
we would like to get rid of. So replace the open coded conversion with
a call to page_to_virt(), and use the open coded conversion as its
default definition, to be overriden by the architecture, if desired.
The existing arch specific definitions of page_to_virt are all equivalent
to this default definition, so by itself this patch is a no-op.
Signed-off-by: Ard Biesheuvel <redacted>
---
include/linux/mm.h | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
From: Will Deacon <hidden> Date: 2016-04-14 15:25:36
On Wed, Mar 30, 2016 at 04:46:04PM +0200, Ard Biesheuvel wrote:
The open coded conversion from struct page address to virtual address in
lowmem_page_address() involves an intermediate conversion step to pfn
number/physical address. Since the placement of the struct page array
relative to the linear mapping may be completely independent from the
placement of physical RAM (as is that case for arm64 after commit
dfd55ad85e 'arm64: vmemmap: use virtual projection of linear region'),
the conversion to physical address and back again should factor out of
the equation, but unfortunately, the shifting and pointer arithmetic
involved prevent this from happening, and the resulting calculation
essentially subtracts the address of the start of physical memory and
adds it back again, in a way that prevents the compiler from optimizing
it away.
Since the start of physical memory is not a build time constant on arm64,
the resulting conversion involves an unnecessary memory access, which
we would like to get rid of. So replace the open coded conversion with
a call to page_to_virt(), and use the open coded conversion as its
default definition, to be overriden by the architecture, if desired.
The existing arch specific definitions of page_to_virt are all equivalent
to this default definition, so by itself this patch is a no-op.
Signed-off-by: Ard Biesheuvel <redacted>
Acked-by: Will Deacon <redacted>
I assume you'll post this patch (and the nios2/openrisc) patches as
individual patches targetting the relevant trees?
Will
On 14 April 2016 at 17:25, Will Deacon [off-list ref] wrote:
On Wed, Mar 30, 2016 at 04:46:04PM +0200, Ard Biesheuvel wrote:
quoted
The open coded conversion from struct page address to virtual address in
lowmem_page_address() involves an intermediate conversion step to pfn
number/physical address. Since the placement of the struct page array
relative to the linear mapping may be completely independent from the
placement of physical RAM (as is that case for arm64 after commit
dfd55ad85e 'arm64: vmemmap: use virtual projection of linear region'),
the conversion to physical address and back again should factor out of
the equation, but unfortunately, the shifting and pointer arithmetic
involved prevent this from happening, and the resulting calculation
essentially subtracts the address of the start of physical memory and
adds it back again, in a way that prevents the compiler from optimizing
it away.
Since the start of physical memory is not a build time constant on arm64,
the resulting conversion involves an unnecessary memory access, which
we would like to get rid of. So replace the open coded conversion with
a call to page_to_virt(), and use the open coded conversion as its
default definition, to be overriden by the architecture, if desired.
The existing arch specific definitions of page_to_virt are all equivalent
to this default definition, so by itself this patch is a no-op.
Signed-off-by: Ard Biesheuvel <redacted>
Acked-by: Will Deacon <redacted>
I assume you'll post this patch (and the nios2/openrisc) patches as
individual patches targetting the relevant trees?
Sure, as they are completely independent from the rest of the series.