From: Andrey Konovalov <redacted>
Hi,
This patchset adds vmalloc tagging support for SW_TAGS and HW_TAGS
KASAN modes.
The tree with patches is available here:
https://github.com/xairy/linux/tree/up-kasan-vmalloc-tags-v5-akpm
About half of patches are cleanups I went for along the way. None of
them seem to be important enough to go through stable, so I decided
not to split them out into separate patches/series.
The patchset is partially based on an early version of the HW_TAGS
patchset by Vincenzo that had vmalloc support. Thus, I added a
Co-developed-by tag into a few patches.
SW_TAGS vmalloc tagging support is straightforward. It reuses all of
the generic KASAN machinery, but uses shadow memory to store tags
instead of magic values. Naturally, vmalloc tagging requires adding
a few kasan_reset_tag() annotations to the vmalloc code.
HW_TAGS vmalloc tagging support stands out. HW_TAGS KASAN is based on
Arm MTE, which can only assigns tags to physical memory. As a result,
HW_TAGS KASAN only tags vmalloc() allocations, which are backed by
page_alloc memory. It ignores vmap() and others.
Thanks!
Changes in v4->v5:
- Rebase onto fresh mm.
- Mention optimization intention in the comment for __GFP_ZEROTAGS.
- Replace "kasan: simplify kasan_init_hw_tags" with "kasan: clean up
feature flags for HW_TAGS mode".
- Use true as kasan_flag_vmalloc static key default.
- Cosmetic changes to __def_gfpflag_names_kasan and __GFP_BITS_SHIFT.
Changes in v3->v4:
- Rebase onto fresh mm.
- Rename KASAN_VMALLOC_NOEXEC to KASAN_VMALLOC_PROT_NORMAL.
- Compare prot with PAGE_KERNEL instead of using pgprot_nx() to
indentify normal non-executable mappings.
- Rename arch_vmalloc_pgprot_modify() to arch_vmap_pgprot_tagged().
- Move checks from arch_vmap_pgprot_tagged() to __vmalloc_node_range()
as the same condition is used for other things in subsequent patches.
- Use proper kasan_hw_tags_enabled() checks instead of
IS_ENABLED(CONFIG_KASAN_HW_TAGS).
- Set __GFP_SKIP_KASAN_UNPOISON and __GFP_SKIP_ZERO flags instead of
resetting.
- Only define KASAN GFP flags when when HW_TAGS KASAN is enabled.
- Move setting KASAN GFP flags to __vmalloc_node_range() and do it
only for normal non-executable mapping when HW_TAGS KASAN is enabled.
- Add new GFP flags to include/trace/events/mmflags.h.
- Don't forget to save tagged addr to vm_struct->addr for VM_ALLOC
so that find_vm_area(addr)->addr == addr for vmalloc().
- Reset pointer tag in change_memory_common().
- Add test checks for set_memory_*() on vmalloc() allocations.
- Minor patch descriptions and comments fixes.
Changes in v2->v3:
- Rebase onto mm.
- New patch: "kasan, arm64: reset pointer tags of vmapped stacks".
- New patch: "kasan, vmalloc: don't tag executable vmalloc allocations".
- New patch: "kasan, arm64: don't tag executable vmalloc allocations".
- Allowing enabling KASAN_VMALLOC with SW/HW_TAGS is moved to
"kasan: allow enabling KASAN_VMALLOC and SW/HW_TAGS", as this can only
be done once executable allocations are no longer tagged.
- Minor fixes, see patches for lists of changes.
Changes in v1->v2:
- Move memory init for vmalloc() into vmalloc code for HW_TAGS KASAN.
- Minor fixes and code reshuffling, see patches for lists of changes.
Acked-by: Marco Elver <elver@google.com>
Andrey Konovalov (39):
kasan, page_alloc: deduplicate should_skip_kasan_poison
kasan, page_alloc: move tag_clear_highpage out of
kernel_init_free_pages
kasan, page_alloc: merge kasan_free_pages into free_pages_prepare
kasan, page_alloc: simplify kasan_poison_pages call site
kasan, page_alloc: init memory of skipped pages on free
kasan: drop skip_kasan_poison variable in free_pages_prepare
mm: clarify __GFP_ZEROTAGS comment
kasan: only apply __GFP_ZEROTAGS when memory is zeroed
kasan, page_alloc: refactor init checks in post_alloc_hook
kasan, page_alloc: merge kasan_alloc_pages into post_alloc_hook
kasan, page_alloc: combine tag_clear_highpage calls in post_alloc_hook
kasan, page_alloc: move SetPageSkipKASanPoison in post_alloc_hook
kasan, page_alloc: move kernel_init_free_pages in post_alloc_hook
kasan, page_alloc: rework kasan_unpoison_pages call site
kasan: clean up metadata byte definitions
kasan: define KASAN_VMALLOC_INVALID for SW_TAGS
kasan, x86, arm64, s390: rename functions for modules shadow
kasan, vmalloc: drop outdated VM_KASAN comment
kasan: reorder vmalloc hooks
kasan: add wrappers for vmalloc hooks
kasan, vmalloc: reset tags in vmalloc functions
kasan, fork: reset pointer tags of vmapped stacks
kasan, arm64: reset pointer tags of vmapped stacks
kasan, vmalloc: add vmalloc tagging for SW_TAGS
kasan, vmalloc, arm64: mark vmalloc mappings as pgprot_tagged
kasan, vmalloc: unpoison VM_ALLOC pages after mapping
kasan, mm: only define ___GFP_SKIP_KASAN_POISON with HW_TAGS
kasan, page_alloc: allow skipping unpoisoning for HW_TAGS
kasan, page_alloc: allow skipping memory init for HW_TAGS
kasan, vmalloc: add vmalloc tagging for HW_TAGS
kasan, vmalloc: only tag normal vmalloc allocations
kasan, arm64: don't tag executable vmalloc allocations
kasan: mark kasan_arg_stacktrace as __initdata
kasan: clean up feature flags for HW_TAGS mode
kasan: add kasan.vmalloc command line flag
kasan: allow enabling KASAN_VMALLOC and SW/HW_TAGS
arm64: select KASAN_VMALLOC for SW/HW_TAGS modes
kasan: documentation updates
kasan: improve vmalloc tests
Documentation/dev-tools/kasan.rst | 17 ++-
arch/arm64/Kconfig | 2 +-
arch/arm64/include/asm/vmalloc.h | 6 +
arch/arm64/include/asm/vmap_stack.h | 5 +-
arch/arm64/kernel/module.c | 5 +-
arch/arm64/mm/pageattr.c | 2 +-
arch/arm64/net/bpf_jit_comp.c | 3 +-
arch/s390/kernel/module.c | 2 +-
arch/x86/kernel/module.c | 2 +-
include/linux/gfp.h | 35 +++--
include/linux/kasan.h | 97 +++++++++-----
include/linux/vmalloc.h | 18 +--
include/trace/events/mmflags.h | 14 +-
kernel/fork.c | 1 +
kernel/scs.c | 4 +-
lib/Kconfig.kasan | 20 +--
lib/test_kasan.c | 189 ++++++++++++++++++++++++++-
mm/kasan/common.c | 4 +-
mm/kasan/hw_tags.c | 193 ++++++++++++++++++++++------
mm/kasan/kasan.h | 18 ++-
mm/kasan/shadow.c | 63 +++++----
mm/page_alloc.c | 157 +++++++++++++++-------
mm/vmalloc.c | 99 +++++++++++---
23 files changed, 736 insertions(+), 220 deletions(-)
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Currently, kernel_init_free_pages() serves two purposes: it either only
zeroes memory or zeroes both memory and memory tags via a different
code path. As this function has only two callers, each using only one
code path, this behaviour is confusing.
Pull the code that zeroes both memory and tags out of
kernel_init_free_pages().
As a result of this change, the code in free_pages_prepare() starts to
look complicated, but this is improved in the few following patches.
Those improvements are not integrated into this patch to make diffs
easier to read.
This patch does no functional changes.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
Changes v2->v3:
- Update patch description.
---
mm/page_alloc.c | 24 +++++++++++++-----------
1 file changed, 13 insertions(+), 11 deletions(-)
From: Andrey Konovalov <redacted>
Currently, the code responsible for initializing and poisoning memory
in free_pages_prepare() is scattered across two locations:
kasan_free_pages() for HW_TAGS KASAN and free_pages_prepare() itself.
This is confusing.
This and a few following patches combine the code from these two
locations. Along the way, these patches also simplify the performed
checks to make them easier to follow.
Replaces the only caller of kasan_free_pages() with its implementation.
As kasan_has_integrated_init() is only true when CONFIG_KASAN_HW_TAGS
is enabled, moving the code does no functional changes.
This patch is not useful by itself but makes the simplifications in
the following patches easier to follow.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
Changes v2->v3:
- Update patch description.
---
include/linux/kasan.h | 8 --------
mm/kasan/common.c | 2 +-
mm/kasan/hw_tags.c | 11 -----------
mm/page_alloc.c | 6 ++++--
4 files changed, 5 insertions(+), 22 deletions(-)
From: Andrey Konovalov <redacted>
Since commit 7a3b83537188 ("kasan: use separate (un)poison implementation
for integrated init"), when all init, kasan_has_integrated_init(), and
skip_kasan_poison are true, free_pages_prepare() doesn't initialize
the page. This is wrong.
Fix it by remembering whether kasan_poison_pages() performed
initialization, and call kernel_init_free_pages() if it didn't.
Reordering kasan_poison_pages() and kernel_init_free_pages() is OK,
since kernel_init_free_pages() can handle poisoned memory.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v2->v3:
- Drop Fixes tag, as the patch won't cleanly apply to older kernels
anyway. The commit is mentioned in the patch description.
Changes v1->v2:
- Reorder kasan_poison_pages() and free_pages_prepare() in this patch
instead of doing it in the previous one.
---
mm/page_alloc.c | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
@@ -1375,11 +1375,16 @@ static __always_inline bool free_pages_prepare(struct page *page,*Withhardwaretag-basedKASAN,memorytagsmustbesetbeforethe*pagebecomesunavailableviadebug_pageallocorarch_free_page.*/-if(init&&!kasan_has_integrated_init())-kernel_init_free_pages(page,1<<order);-if(!skip_kasan_poison)+if(!skip_kasan_poison){kasan_poison_pages(page,order,init);+/* Memory is already initialized if KASAN did it internally. */+if(kasan_has_integrated_init())+init=false;+}+if(init)+kernel_init_free_pages(page,1<<order);+/**arch_free_page()canmakethepage'scontentsinaccessible.s390*doesthis.Sonothingwhichcanaccessthepage'scontentsshould
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Currently, should_skip_kasan_poison() has two definitions: one for when
CONFIG_DEFERRED_STRUCT_PAGE_INIT is enabled, one for when it's not.
Instead of duplicating the checks, add a deferred_pages_enabled()
helper and use it in a single should_skip_kasan_poison() definition.
Also move should_skip_kasan_poison() closer to its caller and clarify
all conditions in the comment.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v2->v3:
- Update patch description.
---
mm/page_alloc.c | 55 +++++++++++++++++++++++++++++--------------------
1 file changed, 33 insertions(+), 22 deletions(-)
@@ -378,25 +378,9 @@ int page_group_by_mobility_disabled __read_mostly;*/staticDEFINE_STATIC_KEY_TRUE(deferred_pages);-/*-*Callingkasan_poison_pages()onlyafterdeferredmemoryinitialization-*hascompleted.Poisoningpagesduringdeferredmemoryinitwillgreatly-*lengthentheprocessandcauseprobleminlargememorysystemsasthe-*deferredpagesinitializationisdonewithinterruptdisabled.-*-*Assumingthattherewillbenoreferencetothosenewlyinitialized-*pagesbeforetheyareeverallocated,thisshouldhavenoeffecton-*KASANmemorytrackingasthepoisonwillbeproperlyinsertedatpage-*allocationtime.Theonlycornercaseiswhenpagesareallocatedby-*on-demandallocationandthenfreedagainbeforethedeferredpages-*initializationisdone,butthisisnotlikelytohappen.-*/-staticinlineboolshould_skip_kasan_poison(structpage*page,fpi_tfpi_flags)+staticinlinebooldeferred_pages_enabled(void){-returnstatic_branch_unlikely(&deferred_pages)||-(!IS_ENABLED(CONFIG_KASAN_GENERIC)&&-(fpi_flags&FPI_SKIP_KASAN_POISON))||-PageSkipKASanPoison(page);+returnstatic_branch_unlikely(&deferred_pages);}/* Returns true if the struct page for the pfn is uninitialised */
@@ -447,11 +431,9 @@ defer_init(int nid, unsigned long pfn, unsigned long end_pfn)returnfalse;}#else-staticinlineboolshould_skip_kasan_poison(structpage*page,fpi_tfpi_flags)+staticinlinebooldeferred_pages_enabled(void){-return(!IS_ENABLED(CONFIG_KASAN_GENERIC)&&-(fpi_flags&FPI_SKIP_KASAN_POISON))||-PageSkipKASanPoison(page);+returnfalse;}staticinlineboolearly_page_uninitialised(unsignedlongpfn)
From: Andrey Konovalov <redacted>
__GFP_ZEROTAGS is intended as an optimization: if memory is zeroed during
allocation, it's possible to set memory tags at the same time with little
performance impact.
Clarify this intention of __GFP_ZEROTAGS in the comment.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v4->v5:
- Mention optimization intention in the comment.
---
include/linux/gfp.h | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
From: Andrey Konovalov <redacted>
skip_kasan_poison is only used in a single place.
Call should_skip_kasan_poison() directly for simplicity.
Signed-off-by: Andrey Konovalov <redacted>
Suggested-by: Marco Elver <elver@google.com>
---
Changes v1->v2:
- Add this patch.
---
mm/page_alloc.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
@@ -1375,7 +1374,7 @@ static __always_inline bool free_pages_prepare(struct page *page,*Withhardwaretag-basedKASAN,memorytagsmustbesetbeforethe*pagebecomesunavailableviadebug_pageallocorarch_free_page.*/-if(!skip_kasan_poison){+if(!should_skip_kasan_poison(page,fpi_flags)){kasan_poison_pages(page,order,init);/* Memory is already initialized if KASAN did it internally. */
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
__GFP_ZEROTAGS should only be effective if memory is being zeroed.
Currently, hardware tag-based KASAN violates this requirement.
Fix by including an initialization check along with checking for
__GFP_ZEROTAGS.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
mm/kasan/hw_tags.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Andrey Konovalov <redacted>
Separate code for zeroing memory from the code clearing tags in
post_alloc_hook().
This patch is not useful by itself but makes the simplifications in
the following patches easier to follow.
This patch does no functional changes.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
Changes v2->v3:
- Update patch description.
---
mm/page_alloc.c | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
From: Andrey Konovalov <redacted>
Currently, the code responsible for initializing and poisoning memory in
post_alloc_hook() is scattered across two locations: kasan_alloc_pages()
hook for HW_TAGS KASAN and post_alloc_hook() itself. This is confusing.
This and a few following patches combine the code from these two
locations. Along the way, these patches do a step-by-step restructure
the many performed checks to make them easier to follow.
Replace the only caller of kasan_alloc_pages() with its implementation.
As kasan_has_integrated_init() is only true when CONFIG_KASAN_HW_TAGS
is enabled, moving the code does no functional changes.
Also move init and init_tags variables definitions out of
kasan_has_integrated_init() clause in post_alloc_hook(), as they have
the same values regardless of what the if condition evaluates to.
This patch is not useful by itself but makes the simplifications in
the following patches easier to follow.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v2->v3:
- Update patch description.
---
include/linux/kasan.h | 9 ---------
mm/kasan/common.c | 2 +-
mm/kasan/hw_tags.c | 22 ----------------------
mm/page_alloc.c | 20 +++++++++++++++-----
4 files changed, 16 insertions(+), 37 deletions(-)
@@ -2419,30 +2419,30 @@ inline void post_alloc_hook(struct page *page, unsigned int order,*KASANunpoisoningandmemoryinitializioncodemustbe*kepttogethertoavoiddiscrepanciesinbehavior.*/++/*+*Ifmemorytagsshouldbezeroed(whichhappensonlywhenmemory+*shouldbeinitializedaswell).+*/+if(init_tags){+inti;++/* Initialize both memory and tags. */+for(i=0;i!=1<<order;++i)+tag_clear_highpage(page+i);++/* Note that memory is already initialized by the loop above. */+init=false;+}if(kasan_has_integrated_init()){if(gfp_flags&__GFP_SKIP_KASAN_POISON)SetPageSkipKASanPoison(page);-if(init_tags){-inti;--for(i=0;i!=1<<order;++i)-tag_clear_highpage(page+i);-}else{+if(!init_tags)kasan_unpoison_pages(page,order,init);-}}else{kasan_unpoison_pages(page,order,init);-if(init_tags){-inti;--for(i=0;i<1<<order;i++)-tag_clear_highpage(page+i);--init=false;-}-if(init)kernel_init_free_pages(page,1<<order);}
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Pull the SetPageSkipKASanPoison() call in post_alloc_hook() out of the
big if clause for better code readability. This also allows for more
simplifications in the following patches.
Also turn the kasan_has_integrated_init() check into the proper
kasan_hw_tags_enabled() one. These checks evaluate to the same value,
but logically skipping kasan poisoning has nothing to do with
integrated init.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v3->v4:
- Use proper kasan_hw_tags_enabled() check instead of
IS_ENABLED(CONFIG_KASAN_HW_TAGS).
---
mm/page_alloc.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
From: Andrey Konovalov <redacted>
Pull the kernel_init_free_pages() call in post_alloc_hook() out of the
big if clause for better code readability. This also allows for more
simplifications in the following patch.
This patch does no functional changes.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
mm/page_alloc.c | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
@@ -2435,14 +2435,18 @@ inline void post_alloc_hook(struct page *page, unsigned int order,init=false;}if(kasan_has_integrated_init()){-if(!init_tags)+if(!init_tags){kasan_unpoison_pages(page,order,init);++/* Note that memory is already initialized by KASAN. */+init=false;+}}else{kasan_unpoison_pages(page,order,init);--if(init)-kernel_init_free_pages(page,1<<order);}+/* If memory is still not initialized, do it now. */+if(init)+kernel_init_free_pages(page,1<<order);/* Propagate __GFP_SKIP_KASAN_POISON to page flags. */if(kasan_hw_tags_enabled()&&(gfp_flags&__GFP_SKIP_KASAN_POISON))SetPageSkipKASanPoison(page);
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Rework the checks around kasan_unpoison_pages() call in
post_alloc_hook().
The logical condition for calling this function is:
- If a software KASAN mode is enabled, we need to mark shadow memory.
- Otherwise, HW_TAGS KASAN is enabled, and it only makes sense to
set tags if they haven't already been cleared by tag_clear_highpage(),
which is indicated by init_tags.
This patch concludes the changes for post_alloc_hook().
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v3->v4:
- Make the confition checks more explicit.
- Update patch description.
---
mm/page_alloc.c | 19 ++++++++++++-------
1 file changed, 12 insertions(+), 7 deletions(-)
@@ -2434,15 +2434,20 @@ inline void post_alloc_hook(struct page *page, unsigned int order,/* Note that memory is already initialized by the loop above. */init=false;}-if(kasan_has_integrated_init()){-if(!init_tags){-kasan_unpoison_pages(page,order,init);+/*+*IfeitherasoftwareKASANmodeisenabled,or,+*inthecaseofhardwaretag-basedKASAN,+*ifmemorytagshavenotbeenclearedviatag_clear_highpage().+*/+if(IS_ENABLED(CONFIG_KASAN_GENERIC)||+IS_ENABLED(CONFIG_KASAN_SW_TAGS)||+kasan_hw_tags_enabled()&&!init_tags){+/* Mark shadow memory or set memory tags. */+kasan_unpoison_pages(page,order,init);-/* Note that memory is already initialized by KASAN. */+/* Note that memory is already initialized by KASAN. */+if(kasan_has_integrated_init())init=false;-}-}else{-kasan_unpoison_pages(page,order,init);}/* If memory is still not initialized, do it now. */if(init)
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Most of the metadata byte values are only used for Generic KASAN.
Remove KASAN_KMALLOC_FREETRACK definition for !CONFIG_KASAN_GENERIC
case, and put it along with other metadata values for the Generic
mode under a corresponding ifdef.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
mm/kasan/kasan.h | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
From: Andrey Konovalov <redacted>
Rename kasan_free_shadow to kasan_free_module_shadow and
kasan_module_alloc to kasan_alloc_module_shadow.
These functions are used to allocate/free shadow memory for kernel
modules when KASAN_VMALLOC is not enabled. The new names better
reflect their purpose.
Also reword the comment next to their declaration to improve clarity.
Signed-off-by: Andrey Konovalov <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/kernel/module.c | 2 +-
arch/s390/kernel/module.c | 2 +-
arch/x86/kernel/module.c | 2 +-
include/linux/kasan.h | 14 +++++++-------
mm/kasan/shadow.c | 4 ++--
mm/vmalloc.c | 2 +-
6 files changed, 13 insertions(+), 13 deletions(-)
From: Andrey Konovalov <redacted>
The comment about VM_KASAN in include/linux/vmalloc.c is outdated.
VM_KASAN is currently only used to mark vm_areas allocated for
kernel modules when CONFIG_KASAN_VMALLOC is disabled.
Drop the comment.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
include/linux/vmalloc.h | 11 -----------
1 file changed, 11 deletions(-)
From: Andrey Konovalov <redacted>
In preparation for adding vmalloc support to SW_TAGS KASAN,
provide a KASAN_VMALLOC_INVALID definition for it.
HW_TAGS KASAN won't be using this value, as it falls back onto
page_alloc for poisoning freed vmalloc() memory.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
mm/kasan/kasan.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Andrey Konovalov <redacted>
Group functions that [de]populate shadow memory for vmalloc.
Group functions that [un]poison memory for vmalloc.
This patch does no functional changes but prepares KASAN code for
adding vmalloc support to HW_TAGS KASAN.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
include/linux/kasan.h | 20 +++++++++-----------
mm/kasan/shadow.c | 43 ++++++++++++++++++++++---------------------
2 files changed, 31 insertions(+), 32 deletions(-)
@@ -345,27 +345,6 @@ int kasan_populate_vmalloc(unsigned long addr, unsigned long size)return0;}-/*-*Poisontheshadowforavmallocregion.Calledaspartofthe-*freeingprocessatthetimetheregionisfreed.-*/-voidkasan_poison_vmalloc(constvoid*start,unsignedlongsize)-{-if(!is_vmalloc_or_module_addr(start))-return;--size=round_up(size,KASAN_GRANULE_SIZE);-kasan_poison(start,size,KASAN_VMALLOC_INVALID,false);-}--voidkasan_unpoison_vmalloc(constvoid*start,unsignedlongsize)-{-if(!is_vmalloc_or_module_addr(start))-return;--kasan_unpoison(start,size,false);-}-staticintkasan_depopulate_vmalloc_pte(pte_t*ptep,unsignedlongaddr,void*unused){
@@ -496,6 +475,28 @@ void kasan_release_vmalloc(unsigned long start, unsigned long end,}}++voidkasan_unpoison_vmalloc(constvoid*start,unsignedlongsize)+{+if(!is_vmalloc_or_module_addr(start))+return;++kasan_unpoison(start,size,false);+}++/*+*Poisontheshadowforavmallocregion.Calledaspartofthe+*freeingprocessatthetimetheregionisfreed.+*/+voidkasan_poison_vmalloc(constvoid*start,unsignedlongsize)+{+if(!is_vmalloc_or_module_addr(start))+return;++size=round_up(size,KASAN_GRANULE_SIZE);+kasan_poison(start,size,KASAN_VMALLOC_INVALID,false);+}+#else /* CONFIG_KASAN_VMALLOC */intkasan_alloc_module_shadow(void*addr,size_tsize,gfp_tgfp_mask)
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Add wrappers around functions that [un]poison memory for vmalloc
allocations. These functions will be used by HW_TAGS KASAN and
therefore need to be disabled when kasan=off command line argument
is provided.
This patch does no functional changes for software KASAN modes.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
include/linux/kasan.h | 17 +++++++++++++++--
mm/kasan/shadow.c | 5 ++---
2 files changed, 17 insertions(+), 5 deletions(-)
From: Andrey Konovalov <redacted>
In preparation for adding vmalloc support to SW/HW_TAGS KASAN,
reset pointer tags in functions that use pointer values in
range checks.
vread() is a special case here. Despite the untagging of the addr
pointer in its prologue, the accesses performed by vread() are checked.
Instead of accessing the virtual mappings though addr directly, vread()
recovers the physical address via page_address(vmalloc_to_page()) and
acceses that. And as page_address() recovers the pointer tag, the
accesses get checked.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v1->v2:
- Clarified the description of untagging in vread().
---
mm/vmalloc.c | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
From: Andrey Konovalov <redacted>
Once tag-based KASAN modes start tagging vmalloc() allocations,
kernel stacks start getting tagged if CONFIG_VMAP_STACK is enabled.
Reset the tag of kernel stack pointers after allocation in
alloc_thread_stack_node().
For SW_TAGS KASAN, when CONFIG_KASAN_STACK is enabled, the
instrumentation can't handle the SP register being tagged.
For HW_TAGS KASAN, there's no instrumentation-related issues. However,
the impact of having a tagged SP register needs to be properly evaluated,
so keep it non-tagged for now.
Note, that the memory for the stack allocation still gets tagged to
catch vmalloc-into-stack out-of-bounds accesses.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
Changes v2->v3:
- Update patch description.
---
kernel/fork.c | 1 +
1 file changed, 1 insertion(+)
From: Andrey Konovalov <redacted>
Once tag-based KASAN modes start tagging vmalloc() allocations,
kernel stacks start getting tagged if CONFIG_VMAP_STACK is enabled.
Reset the tag of kernel stack pointers after allocation in
arch_alloc_vmap_stack().
For SW_TAGS KASAN, when CONFIG_KASAN_STACK is enabled, the
instrumentation can't handle the SP register being tagged.
For HW_TAGS KASAN, there's no instrumentation-related issues. However,
the impact of having a tagged SP register needs to be properly evaluated,
so keep it non-tagged for now.
Note, that the memory for the stack allocation still gets tagged to
catch vmalloc-into-stack out-of-bounds accesses.
Signed-off-by: Andrey Konovalov <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
---
Changes v2->v3:
- Add this patch.
---
arch/arm64/include/asm/vmap_stack.h | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
From: Andrey Konovalov <redacted>
Add vmalloc tagging support to SW_TAGS KASAN.
- __kasan_unpoison_vmalloc() now assigns a random pointer tag, poisons
the virtual mapping accordingly, and embeds the tag into the returned
pointer.
- __get_vm_area_node() (used by vmalloc() and vmap()) and
pcpu_get_vm_areas() save the tagged pointer into vm_struct->addr
(note: not into vmap_area->addr). This requires putting
kasan_unpoison_vmalloc() after setup_vmalloc_vm[_locked]();
otherwise the latter will overwrite the tagged pointer.
The tagged pointer then is naturally propagateed to vmalloc()
and vmap().
- vm_map_ram() returns the tagged pointer directly.
As a result of this change, vm_struct->addr is now tagged.
Enabling KASAN_VMALLOC with SW_TAGS is not yet allowed.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v2->v3:
- Drop accidentally added kasan_unpoison_vmalloc() argument for when
KASAN is off.
- Drop __must_check for kasan_unpoison_vmalloc(), as its result is
sometimes intentionally ignored.
- Move allowing enabling KASAN_VMALLOC with SW_TAGS into a separate
patch.
- Update patch description.
Changes v1->v2:
- Allow enabling KASAN_VMALLOC with SW_TAGS in this patch.
---
include/linux/kasan.h | 16 ++++++++++------
mm/kasan/shadow.c | 6 ++++--
mm/vmalloc.c | 14 ++++++++------
3 files changed, 22 insertions(+), 14 deletions(-)
@@ -2210,7 +2210,7 @@ void *vm_map_ram(struct page **pages, unsigned int count, int node)mem=(void*)addr;}-kasan_unpoison_vmalloc(mem,size);+mem=kasan_unpoison_vmalloc(mem,size);if(vmap_pages_range(addr,addr+size,PAGE_KERNEL,pages,PAGE_SHIFT)<0){
@@ -2443,10 +2443,10 @@ static struct vm_struct *__get_vm_area_node(unsigned long size,returnNULL;}-kasan_unpoison_vmalloc((void*)va->va_start,requested_size);-setup_vmalloc_vm(area,va,flags,caller);+area->addr=kasan_unpoison_vmalloc(area->addr,requested_size);+returnarea;}
@@ -3802,9 +3802,6 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,for(area=0;area<nr_vms;area++){if(kasan_populate_vmalloc(vas[area]->va_start,sizes[area]))gotoerr_free_shadow;--kasan_unpoison_vmalloc((void*)vas[area]->va_start,-sizes[area]);}/* insert all vm's */
@@ -3817,6 +3814,11 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,}spin_unlock(&vmap_area_lock);+/* mark allocated areas as accessible */+for(area=0;area<nr_vms;area++)+vms[area]->addr=kasan_unpoison_vmalloc(vms[area]->addr,+vms[area]->size);+kfree(vas);returnvms;
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
HW_TAGS KASAN relies on ARM Memory Tagging Extension (MTE). With MTE,
a memory region must be mapped as MT_NORMAL_TAGGED to allow setting
memory tags via MTE-specific instructions.
Add proper protection bits to vmalloc() allocations. These allocations
are always backed by page_alloc pages, so the tags will actually be
getting set on the corresponding physical memory.
Signed-off-by: Andrey Konovalov <redacted>
Co-developed-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
---
Changes v3->v4:
- Rename arch_vmalloc_pgprot_modify() to arch_vmap_pgprot_tagged()
to be consistent with other arch vmalloc hooks.
- Move checks from arch_vmap_pgprot_tagged() to __vmalloc_node_range()
as the same condition is used for other things in subsequent patches.
Changes v2->v3:
- Update patch description.
---
arch/arm64/include/asm/vmalloc.h | 6 ++++++
include/linux/vmalloc.h | 7 +++++++
mm/vmalloc.c | 9 +++++++++
3 files changed, 22 insertions(+)
@@ -3107,6 +3107,15 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align,gotofail;}+/*+*Modifyprotectionbitstoallowtagging.+*Thismustbedonebeforemappingby__vmalloc_area_node().+*/+if(kasan_hw_tags_enabled()&&+pgprot_val(prot)==pgprot_val(PAGE_KERNEL))+prot=arch_vmap_pgprot_tagged(prot);++/* Allocate physical pages and map them into vmalloc space. */addr=__vmalloc_area_node(area,gfp_mask,prot,shift,node);if(!addr)gotofail;
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Add a new GFP flag __GFP_SKIP_KASAN_UNPOISON that allows skipping KASAN
poisoning for page_alloc allocations. The flag is only effective with
HW_TAGS KASAN.
This flag will be used by vmalloc code for page_alloc allocations
backing vmalloc() mappings in a following patch. The reason to skip
KASAN poisoning for these pages in page_alloc is because vmalloc code
will be poisoning them instead.
Also reword the comment for __GFP_SKIP_KASAN_POISON.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v4->v5:
- Cosmetic changes to __def_gfpflag_names_kasan and __GFP_BITS_SHIFT.
Changes v3->v4:
- Only define __GFP_SKIP_KASAN_POISON when CONFIG_KASAN_HW_TAGS is
enabled.
Changes v2->v3:
- Update patch description.
Signed-off-by: Andrey Konovalov <redacted>
---
include/linux/gfp.h | 21 +++++++++++++--------
include/trace/events/mmflags.h | 5 +++--
mm/page_alloc.c | 31 ++++++++++++++++++++++---------
3 files changed, 38 insertions(+), 19 deletions(-)
@@ -2395,6 +2395,26 @@ static bool check_new_pages(struct page *page, unsigned int order)returnfalse;}+staticinlineboolshould_skip_kasan_unpoison(gfp_tflags,boolinit_tags)+{+/* Don't skip if a software KASAN mode is enabled. */+if(IS_ENABLED(CONFIG_KASAN_GENERIC)||+IS_ENABLED(CONFIG_KASAN_SW_TAGS))+returnfalse;++/* Skip, if hardware tag-based KASAN is not enabled. */+if(!kasan_hw_tags_enabled())+returntrue;++/*+*Withhardwaretag-basedKASANenabled,skipifeither:+*+*1.Memorytagshavealreadybeenclearedviatag_clear_highpage().+*2.Skippinghasbeenrequestedvia__GFP_SKIP_KASAN_UNPOISON.+*/+returninit_tags||(flags&__GFP_SKIP_KASAN_UNPOISON);+}+inlinevoidpost_alloc_hook(structpage*page,unsignedintorder,gfp_tgfp_flags){
@@ -2434,15 +2454,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order,/* Note that memory is already initialized by the loop above. */init=false;}-/*-*IfeitherasoftwareKASANmodeisenabled,or,-*inthecaseofhardwaretag-basedKASAN,-*ifmemorytagshavenotbeenclearedviatag_clear_highpage().-*/-if(IS_ENABLED(CONFIG_KASAN_GENERIC)||-IS_ENABLED(CONFIG_KASAN_SW_TAGS)||-kasan_hw_tags_enabled()&&!init_tags){-/* Mark shadow memory or set memory tags. */+if(!should_skip_kasan_unpoison(gfp_flags,init_tags)){+/* Unpoison shadow memory or set memory tags. */kasan_unpoison_pages(page,order,init);/* Note that memory is already initialized by KASAN. */
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Only define the ___GFP_SKIP_KASAN_POISON flag when CONFIG_KASAN_HW_TAGS
is enabled.
This patch it not useful by itself, but it prepares the code for
additions of new KASAN-specific GFP patches.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v3->v4:
- This is a new patch.
---
include/linux/gfp.h | 8 +++++++-
include/trace/events/mmflags.h | 12 +++++++++---
2 files changed, 16 insertions(+), 4 deletions(-)
From: Andrey Konovalov <redacted>
Make KASAN unpoison vmalloc mappings after they have been mapped in
when it's possible: for vmalloc() (indentified via VM_ALLOC) and
vm_map_ram().
The reasons for this are:
- For vmalloc() and vm_map_ram(): pages don't get unpoisoned in case
mapping them fails.
- For vmalloc(): HW_TAGS KASAN needs pages to be mapped to set tags via
kasan_unpoison_vmalloc().
As a part of these changes, the return value of __vmalloc_node_range()
is changed to area->addr. This is a non-functional change, as
__vmalloc_area_node() returns area->addr anyway.
Signed-off-by: Andrey Konovalov <redacted>
Reviewed-by: Alexander Potapenko <glider@google.com>
---
Changes v3->v4:
- Don't forget to save tagged addr to vm_struct->addr for VM_ALLOC
so that find_vm_area(addr)->addr == addr for vmalloc().
- Reword comments.
- Update patch description.
Changes v2->v3:
- Update patch description.
---
mm/vmalloc.c | 30 ++++++++++++++++++++++--------
1 file changed, 22 insertions(+), 8 deletions(-)
@@ -2210,14 +2210,15 @@ void *vm_map_ram(struct page **pages, unsigned int count, int node)mem=(void*)addr;}-mem=kasan_unpoison_vmalloc(mem,size);-if(vmap_pages_range(addr,addr+size,PAGE_KERNEL,pages,PAGE_SHIFT)<0){vm_unmap_ram(mem,count);returnNULL;}+/* Mark the pages as accessible, now that they are mapped. */+mem=kasan_unpoison_vmalloc(mem,size);+returnmem;}EXPORT_SYMBOL(vm_map_ram);
@@ -2445,7 +2446,14 @@ static struct vm_struct *__get_vm_area_node(unsigned long size,setup_vmalloc_vm(area,va,flags,caller);-area->addr=kasan_unpoison_vmalloc(area->addr,requested_size);+/*+*Markpagesfornon-VM_ALLOCmappingsasaccessible.Doitnowasa+*best-effortapproach,astheycanbemappedoutsideofvmalloccode.+*ForVM_ALLOCmappings,thepagesaremarkedasaccessibleafter+*gettingmappedin__vmalloc_node_range().+*/+if(!(flags&VM_ALLOC))+area->addr=kasan_unpoison_vmalloc(area->addr,requested_size);returnarea;}
@@ -3054,7 +3062,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align,constvoid*caller){structvm_struct*area;-void*addr;+void*ret;unsignedlongreal_size=size;unsignedlongreal_align=align;unsignedintshift=PAGE_SHIFT;
@@ -3116,10 +3124,13 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align,prot=arch_vmap_pgprot_tagged(prot);/* Allocate physical pages and map them into vmalloc space. */-addr=__vmalloc_area_node(area,gfp_mask,prot,shift,node);-if(!addr)+ret=__vmalloc_area_node(area,gfp_mask,prot,shift,node);+if(!ret)gotofail;+/* Mark the pages as accessible, now that they are mapped. */+area->addr=kasan_unpoison_vmalloc(area->addr,real_size);+/**Inthisfunction,newlyallocatedvm_structhasVM_UNINITIALIZED*flag.Itmeansthatvm_structisnotfullyinitialized.
@@ -3131,7 +3142,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align,if(!(vm_flags&VM_DEFER_KMEMLEAK))kmemleak_vmalloc(area,size,gfp_mask);-returnaddr;+returnarea->addr;fail:if(shift>PAGE_SHIFT){
@@ -3823,7 +3834,10 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,}spin_unlock(&vmap_area_lock);-/* mark allocated areas as accessible */+/*+*Markallocatedareasasaccessible.Doitnowasabest-effort+*approach,astheycanbemappedoutsideofvmalloccode.+*/for(area=0;area<nr_vms;area++)vms[area]->addr=kasan_unpoison_vmalloc(vms[area]->addr,vms[area]->size);
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Add a new GFP flag __GFP_SKIP_ZERO that allows to skip memory
initialization. The flag is only effective with HW_TAGS KASAN.
This flag will be used by vmalloc code for page_alloc allocations
backing vmalloc() mappings in a following patch. The reason to skip
memory initialization for these pages in page_alloc is because vmalloc
code will be initializing them instead.
With the current implementation, when __GFP_SKIP_ZERO is provided,
__GFP_ZEROTAGS is ignored. This doesn't matter, as these two flags are
never provided at the same time. However, if this is changed in the
future, this particular implementation detail can be changed as well.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v4->v5:
- Cosmetic changes to __def_gfpflag_names_kasan and __GFP_BITS_SHIFT.
Changes v3->v4:
- Only define __GFP_SKIP_ZERO when CONFIG_KASAN_HW_TAGS is enabled.
- Add __GFP_SKIP_ZERO to include/trace/events/mmflags.h.
- Use proper kasan_hw_tags_enabled() check instead of
IS_ENABLED(CONFIG_KASAN_HW_TAGS). Also add explicit checks for
software modes.
Changes v2->v3:
- Update patch description.
Changes v1->v2:
- Add this patch.
---
include/linux/gfp.h | 18 +++++++++++-------
include/trace/events/mmflags.h | 1 +
mm/page_alloc.c | 18 +++++++++++++++++-
3 files changed, 29 insertions(+), 8 deletions(-)
@@ -2415,10 +2415,26 @@ static inline bool should_skip_kasan_unpoison(gfp_t flags, bool init_tags)returninit_tags||(flags&__GFP_SKIP_KASAN_UNPOISON);}+staticinlineboolshould_skip_init(gfp_tflags)+{+/* Don't skip if a software KASAN mode is enabled. */+if(IS_ENABLED(CONFIG_KASAN_GENERIC)||+IS_ENABLED(CONFIG_KASAN_SW_TAGS))+returnfalse;++/* Don't skip, if hardware tag-based KASAN is not enabled. */+if(!kasan_hw_tags_enabled())+returnfalse;++/* For hardware tag-based KASAN, skip if requested. */+return(flags&__GFP_SKIP_ZERO);+}+inlinevoidpost_alloc_hook(structpage*page,unsignedintorder,gfp_tgfp_flags){-boolinit=!want_init_on_free()&&want_init_on_alloc(gfp_flags);+boolinit=!want_init_on_free()&&want_init_on_alloc(gfp_flags)&&+!should_skip_init(gfp_flags);boolinit_tags=init&&(gfp_flags&__GFP_ZEROTAGS);set_page_private(page,0);
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Add vmalloc tagging support to HW_TAGS KASAN.
The key difference between HW_TAGS and the other two KASAN modes
when it comes to vmalloc: HW_TAGS KASAN can only assign tags to
physical memory. The other two modes have shadow memory covering
every mapped virtual memory region.
Make __kasan_unpoison_vmalloc() for HW_TAGS KASAN:
- Skip non-VM_ALLOC mappings as HW_TAGS KASAN can only tag a single
mapping of normal physical memory; see the comment in the function.
- Generate a random tag, tag the returned pointer and the allocation,
and initialize the allocation at the same time.
- Propagate the tag into the page stucts to allow accesses through
page_address(vmalloc_to_page()).
The rest of vmalloc-related KASAN hooks are not needed:
- The shadow-related ones are fully skipped.
- __kasan_poison_vmalloc() is kept as a no-op with a comment.
Poisoning and zeroing of physical pages that are backing vmalloc()
allocations are skipped via __GFP_SKIP_KASAN_UNPOISON and
__GFP_SKIP_ZERO: __kasan_unpoison_vmalloc() does that instead.
Enabling CONFIG_KASAN_VMALLOC with HW_TAGS is not yet allowed.
Signed-off-by: Andrey Konovalov <redacted>
Co-developed-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@arm.com>
---
Changes v3->v4:
- Fix comment style in __kasan_unpoison_vmalloc().
- Set __GFP_SKIP_KASAN_UNPOISON and __GFP_SKIP_ZERO flags instead of
resetting.
- Move setting KASAN GFP flags to __vmalloc_node_range() and do it
only for normal non-executable mapping when HW_TAGS KASAN is enabled.
Changes v2->v3:
- Switch kasan_unpoison_vmalloc() to using a single flags argument.
- Update kasan_unpoison_vmalloc() arguments in kernel/scs.c.
- Move allowing enabling KASAN_VMALLOC with SW_TAGS into a separate
patch.
- Minor comments fixes.
- Update patch description.
Changes v1->v2:
- Allow enabling CONFIG_KASAN_VMALLOC with HW_TAGS in this patch.
- Move memory init for page_alloc pages backing vmalloc() into
kasan_unpoison_vmalloc().
---
include/linux/kasan.h | 36 +++++++++++++++--
kernel/scs.c | 4 +-
mm/kasan/hw_tags.c | 92 +++++++++++++++++++++++++++++++++++++++++++
mm/kasan/shadow.c | 10 ++++-
mm/vmalloc.c | 51 ++++++++++++++++++------
5 files changed, 175 insertions(+), 18 deletions(-)
@@ -2216,8 +2216,12 @@ void *vm_map_ram(struct page **pages, unsigned int count, int node)returnNULL;}-/* Mark the pages as accessible, now that they are mapped. */-mem=kasan_unpoison_vmalloc(mem,size);+/*+*Markthepagesasaccessible,nowthattheyaremapped.+*Withhardwaretag-basedKASAN,markingisskippedfor+*non-VM_ALLOCmappings,see__kasan_unpoison_vmalloc().+*/+mem=kasan_unpoison_vmalloc(mem,size,KASAN_VMALLOC_NONE);returnmem;}
@@ -2451,9 +2455,12 @@ static struct vm_struct *__get_vm_area_node(unsigned long size,*best-effortapproach,astheycanbemappedoutsideofvmalloccode.*ForVM_ALLOCmappings,thepagesaremarkedasaccessibleafter*gettingmappedin__vmalloc_node_range().+*Withhardwaretag-basedKASAN,markingisskippedfor+*non-VM_ALLOCmappings,see__kasan_unpoison_vmalloc().*/if(!(flags&VM_ALLOC))-area->addr=kasan_unpoison_vmalloc(area->addr,requested_size);+area->addr=kasan_unpoison_vmalloc(area->addr,requested_size,+KASAN_VMALLOC_NONE);returnarea;}
@@ -3063,6 +3070,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align,{structvm_struct*area;void*ret;+kasan_vmalloc_flags_tkasan_flags;unsignedlongreal_size=size;unsignedlongreal_align=align;unsignedintshift=PAGE_SHIFT;
@@ -3115,21 +3123,39 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align,gotofail;}-/*-*Modifyprotectionbitstoallowtagging.-*Thismustbedonebeforemappingby__vmalloc_area_node().-*/+/* Prepare arguments for __vmalloc_area_node(). */if(kasan_hw_tags_enabled()&&-pgprot_val(prot)==pgprot_val(PAGE_KERNEL))+pgprot_val(prot)==pgprot_val(PAGE_KERNEL)){+/*+*Modifyprotectionbitstoallowtagging.+*Thismustbedonebeforemappingin__vmalloc_area_node().+*/prot=arch_vmap_pgprot_tagged(prot);+/*+*Skippage_allocpoisoningandzeroingforphysicalpages+*backingVM_ALLOCmapping.Memoryisinsteadpoisonedand+*zeroedbykasan_unpoison_vmalloc().+*/+gfp_mask|=__GFP_SKIP_KASAN_UNPOISON|__GFP_SKIP_ZERO;+}+/* Allocate physical pages and map them into vmalloc space. */ret=__vmalloc_area_node(area,gfp_mask,prot,shift,node);if(!ret)gotofail;-/* Mark the pages as accessible, now that they are mapped. */-area->addr=kasan_unpoison_vmalloc(area->addr,real_size);+/*+*Markthepagesasaccessible,nowthattheyaremapped.+*Theinitconditionshouldmatchtheoneinpost_alloc_hook()+*(exceptfortheshould_skip_init()check)tomakesurethatmemory+*isinitializedunderthesameconditionsregardlessoftheenabled+*KASANmode.+*/+kasan_flags=KASAN_VMALLOC_VM_ALLOC;+if(!want_init_on_free()&&want_init_on_alloc(gfp_mask))+kasan_flags|=KASAN_VMALLOC_INIT;+area->addr=kasan_unpoison_vmalloc(area->addr,real_size,kasan_flags);/**Inthisfunction,newlyallocatedvm_structhasVM_UNINITIALIZED
@@ -3837,10 +3863,13 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,/**Markallocatedareasasaccessible.Doitnowasabest-effort*approach,astheycanbemappedoutsideofvmalloccode.+*Withhardwaretag-basedKASAN,markingisskippedfor+*non-VM_ALLOCmappings,see__kasan_unpoison_vmalloc().*/for(area=0;area<nr_vms;area++)vms[area]->addr=kasan_unpoison_vmalloc(vms[area]->addr,-vms[area]->size);+vms[area]->size,+KASAN_VMALLOC_NONE);kfree(vas);returnvms;
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
The kernel can use to allocate executable memory. The only supported way
to do that is via __vmalloc_node_range() with the executable bit set in
the prot argument. (vmap() resets the bit via pgprot_nx()).
Once tag-based KASAN modes start tagging vmalloc allocations, executing
code from such allocations will lead to the PC register getting a tag,
which is not tolerated by the kernel.
Only tag the allocations for normal kernel pages.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v3->v4:
- Rename KASAN_VMALLOC_NOEXEC to KASAN_VMALLOC_PROT_NORMAL.
- Compare with PAGE_KERNEL instead of using pgprot_nx().
- Update patch description.
Changes v2->v3:
- Add this patch.
---
include/linux/kasan.h | 7 ++++---
mm/kasan/hw_tags.c | 7 +++++++
mm/kasan/shadow.c | 7 +++++++
mm/vmalloc.c | 49 +++++++++++++++++++++++++------------------
4 files changed, 47 insertions(+), 23 deletions(-)
@@ -2221,7 +2221,7 @@ void *vm_map_ram(struct page **pages, unsigned int count, int node)*Withhardwaretag-basedKASAN,markingisskippedfor*non-VM_ALLOCmappings,see__kasan_unpoison_vmalloc().*/-mem=kasan_unpoison_vmalloc(mem,size,KASAN_VMALLOC_NONE);+mem=kasan_unpoison_vmalloc(mem,size,KASAN_VMALLOC_PROT_NORMAL);returnmem;}
@@ -2460,7 +2460,7 @@ static struct vm_struct *__get_vm_area_node(unsigned long size,*/if(!(flags&VM_ALLOC))area->addr=kasan_unpoison_vmalloc(area->addr,requested_size,-KASAN_VMALLOC_NONE);+KASAN_VMALLOC_PROT_NORMAL);returnarea;}
@@ -3070,7 +3070,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align,{structvm_struct*area;void*ret;-kasan_vmalloc_flags_tkasan_flags;+kasan_vmalloc_flags_tkasan_flags=KASAN_VMALLOC_NONE;unsignedlongreal_size=size;unsignedlongreal_align=align;unsignedintshift=PAGE_SHIFT;
@@ -3123,21 +3123,28 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align,gotofail;}-/* Prepare arguments for __vmalloc_area_node(). */-if(kasan_hw_tags_enabled()&&-pgprot_val(prot)==pgprot_val(PAGE_KERNEL)){-/*-*Modifyprotectionbitstoallowtagging.-*Thismustbedonebeforemappingin__vmalloc_area_node().-*/-prot=arch_vmap_pgprot_tagged(prot);+/*+*Prepareargumentsfor__vmalloc_area_node()and+*kasan_unpoison_vmalloc().+*/+if(pgprot_val(prot)==pgprot_val(PAGE_KERNEL)){+if(kasan_hw_tags_enabled()){+/*+*Modifyprotectionbitstoallowtagging.+*Thismustbedonebeforemapping.+*/+prot=arch_vmap_pgprot_tagged(prot);-/*-*Skippage_allocpoisoningandzeroingforphysicalpages-*backingVM_ALLOCmapping.Memoryisinsteadpoisonedand-*zeroedbykasan_unpoison_vmalloc().-*/-gfp_mask|=__GFP_SKIP_KASAN_UNPOISON|__GFP_SKIP_ZERO;+/*+*Skippage_allocpoisoningandzeroingforphysical+*pagesbackingVM_ALLOCmapping.Memoryisinstead+*poisonedandzeroedbykasan_unpoison_vmalloc().+*/+gfp_mask|=__GFP_SKIP_KASAN_UNPOISON|__GFP_SKIP_ZERO;+}++/* Take note that the mapping is PAGE_KERNEL. */+kasan_flags|=KASAN_VMALLOC_PROT_NORMAL;}/* Allocate physical pages and map them into vmalloc space. */
@@ -3151,10 +3158,13 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align,*(exceptfortheshould_skip_init()check)tomakesurethatmemory*isinitializedunderthesameconditionsregardlessoftheenabled*KASANmode.+*Tag-basedKASANmodesonlyassigntagstonormalnon-executable+*allocations,see__kasan_unpoison_vmalloc().*/-kasan_flags=KASAN_VMALLOC_VM_ALLOC;+kasan_flags|=KASAN_VMALLOC_VM_ALLOC;if(!want_init_on_free()&&want_init_on_alloc(gfp_mask))kasan_flags|=KASAN_VMALLOC_INIT;+/* KASAN_VMALLOC_PROT_NORMAL already set if required. */area->addr=kasan_unpoison_vmalloc(area->addr,real_size,kasan_flags);/*
@@ -3868,8 +3878,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,*/for(area=0;area<nr_vms;area++)vms[area]->addr=kasan_unpoison_vmalloc(vms[area]->addr,-vms[area]->size,-KASAN_VMALLOC_NONE);+vms[area]->size,KASAN_VMALLOC_PROT_NORMAL);kfree(vas);returnvms;
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Besides asking vmalloc memory to be executable via the prot argument
of __vmalloc_node_range() (see the previous patch), the kernel can skip
that bit and instead mark memory as executable via set_memory_x().
Once tag-based KASAN modes start tagging vmalloc allocations, executing
code from such allocations will lead to the PC register getting a tag,
which is not tolerated by the kernel.
Generic kernel code typically allocates memory via module_alloc() if
it intends to mark memory as executable. (On arm64 module_alloc()
uses __vmalloc_node_range() without setting the executable bit).
Thus, reset pointer tags of pointers returned from module_alloc().
However, on arm64 there's an exception: the eBPF subsystem. Instead of
using module_alloc(), it uses vmalloc() (via bpf_jit_alloc_exec())
to allocate its JIT region.
Thus, reset pointer tags of pointers returned from bpf_jit_alloc_exec().
Resetting tags for these pointers results in untagged pointers being
passed to set_memory_x(). This causes conflicts in arithmetic checks
in change_memory_common(), as vm_struct->addr pointer returned by
find_vm_area() is tagged.
Reset pointer tag of find_vm_area(addr)->addr in change_memory_common().
Signed-off-by: Andrey Konovalov <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
---
Changes v3->v4:
- Reset pointer tag in change_memory_common().
Changes v2->v3:
- Add this patch.
---
arch/arm64/kernel/module.c | 3 ++-
arch/arm64/mm/pageattr.c | 2 +-
arch/arm64/net/bpf_jit_comp.c | 3 ++-
3 files changed, 5 insertions(+), 3 deletions(-)
@@ -63,7 +63,8 @@ void *module_alloc(unsigned long size)returnNULL;}-returnp;+/* Memory is intended to be executable, reset the pointer tag. */+returnkasan_reset_tag(p);}enumaarch64_reloc_op{
@@ -85,7 +85,7 @@ static int change_memory_common(unsigned long addr, int numpages,*/area=find_vm_area((void*)addr);if(!area||-end>(unsignedlong)area->addr+area->size||+end>(unsignedlong)kasan_reset_tag(area->addr)+area->size||!(area->flags&VM_ALLOC))return-EINVAL;
@@ -1147,7 +1147,8 @@ u64 bpf_jit_alloc_exec_limit(void)void*bpf_jit_alloc_exec(unsignedlongsize){-returnvmalloc(size);+/* Memory is intended to be executable, reset the pointer tag. */+returnkasan_reset_tag(vmalloc(size));}voidbpf_jit_free_exec(void*addr)
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
As kasan_arg_stacktrace is only used in __init functions, mark it as
__initdata instead of __ro_after_init to allow it be freed after boot.
The other enums for KASAN args are used in kasan_init_hw_tags_cpu(),
which is not marked as __init as a CPU can be hot-plugged after boot.
Clarify this in a comment.
Signed-off-by: Andrey Konovalov <redacted>
Suggested-by: Marco Elver <elver@google.com>
---
Changes v1->v2:
- Add this patch.
---
mm/kasan/hw_tags.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
@@ -40,7 +40,7 @@ enum kasan_arg_stacktrace {staticenumkasan_argkasan_arg__ro_after_init;staticenumkasan_arg_modekasan_arg_mode__ro_after_init;-staticenumkasan_arg_stacktracekasan_arg_stacktrace__ro_after_init;+staticenumkasan_arg_stacktracekasan_arg_stacktrace__initdata;/* Whether KASAN is enabled at all. */DEFINE_STATIC_KEY_FALSE(kasan_flag_enabled);
@@ -116,7 +116,10 @@ static inline const char *kasan_mode_info(void)return"sync";}-/* kasan_init_hw_tags_cpu() is called for each CPU. */+/*+*kasan_init_hw_tags_cpu()iscalledforeachCPU.+*Notmarkedas__initasaCPUcanbehot-pluggedafterboot.+*/voidkasan_init_hw_tags_cpu(void){/*
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
@@ -42,16 +42,22 @@ static enum kasan_arg kasan_arg __ro_after_init;staticenumkasan_arg_modekasan_arg_mode__ro_after_init;staticenumkasan_arg_stacktracekasan_arg_stacktrace__initdata;-/* Whether KASAN is enabled at all. */+/*+*WhetherKASANisenabledatall.+*ThevalueremainsfalseuntilKASANisinitializedbykasan_init_hw_tags().+*/DEFINE_STATIC_KEY_FALSE(kasan_flag_enabled);EXPORT_SYMBOL(kasan_flag_enabled);-/* Whether the selected mode is synchronous/asynchronous/asymmetric.*/+/*+*Whethertheselectedmodeissynchronous,asynchronous,orasymmetric.+*DefaultstoKASAN_MODE_SYNC.+*/enumkasan_modekasan_mode__ro_after_init;EXPORT_SYMBOL_GPL(kasan_mode);/* Whether to collect alloc/free stack traces. */-DEFINE_STATIC_KEY_FALSE(kasan_flag_stacktrace);+DEFINE_STATIC_KEY_TRUE(kasan_flag_stacktrace);/* kasan=off/on */staticint__initearly_kasan_flag(char*arg)
@@ -127,7 +133,11 @@ void kasan_init_hw_tags_cpu(void)*asthisfunctionisonlycalledforMTE-capablehardware.*/-/* If KASAN is disabled via command line, don't initialize it. */+/*+*IfKASANisdisabledviacommandline,don'tinitializeit.+*Whenthisfunctioniscalled,kasan_flag_enabledisnotyet+*setbykasan_init_hw_tags().Thus,checkkasan_arginstead.+*/if(kasan_arg==KASAN_ARG_OFF)return;
@@ -154,42 +164,36 @@ void __init kasan_init_hw_tags(void)if(kasan_arg==KASAN_ARG_OFF)return;-/* Enable KASAN. */-static_branch_enable(&kasan_flag_enabled);-switch(kasan_arg_mode){caseKASAN_ARG_MODE_DEFAULT:-/*-*Defaulttosyncmode.-*/-fallthrough;+/* Default is specified by kasan_mode definition. */+break;caseKASAN_ARG_MODE_SYNC:-/* Sync mode enabled. */kasan_mode=KASAN_MODE_SYNC;break;caseKASAN_ARG_MODE_ASYNC:-/* Async mode enabled. */kasan_mode=KASAN_MODE_ASYNC;break;caseKASAN_ARG_MODE_ASYMM:-/* Asymm mode enabled. */kasan_mode=KASAN_MODE_ASYMM;break;}switch(kasan_arg_stacktrace){caseKASAN_ARG_STACKTRACE_DEFAULT:-/* Default to enabling stack trace collection. */-static_branch_enable(&kasan_flag_stacktrace);+/* Default is specified by kasan_flag_stacktrace definition. */break;caseKASAN_ARG_STACKTRACE_OFF:-/* Do nothing, kasan_flag_stacktrace keeps its default value. */+static_branch_disable(&kasan_flag_stacktrace);break;caseKASAN_ARG_STACKTRACE_ON:static_branch_enable(&kasan_flag_stacktrace);break;}+/* KASAN is now initialized, enable it. */+static_branch_enable(&kasan_flag_enabled);+pr_info("KernelAddressSanitizer initialized (hw-tags, mode=%s, stacktrace=%s)\n",kasan_mode_info(),kasan_stack_collection_enabled()?"on":"off");
From: Andrey Konovalov <redacted>
Allow disabling vmalloc() tagging for HW_TAGS KASAN via a kasan.vmalloc
command line switch.
This is a fail-safe switch intended for production systems that enable
HW_TAGS KASAN. In case vmalloc() tagging ends up having an issue not
detected during testing but that manifests in production, kasan.vmalloc
allows to turn vmalloc() tagging off while leaving page_alloc/slab
tagging on.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v4->v5:
- Use true as kasan_flag_vmalloc static key default.
Changes v1->v2:
- Mark kasan_arg_stacktrace as __initdata instead of __ro_after_init.
- Combine KASAN_ARG_VMALLOC_DEFAULT and KASAN_ARG_VMALLOC_ON switch
cases.
---
mm/kasan/hw_tags.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
mm/kasan/kasan.h | 6 ++++++
2 files changed, 50 insertions(+), 1 deletion(-)
From: Andrey Konovalov <redacted>
Allow enabling CONFIG_KASAN_VMALLOC with SW_TAGS and HW_TAGS KASAN
modes.
Also adjust CONFIG_KASAN_VMALLOC description:
- Mention HW_TAGS support.
- Remove unneeded internal details: they have no place in Kconfig
description and are already explained in the documentation.
Signed-off-by: Andrey Konovalov <redacted>
---
lib/Kconfig.kasan | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
@@ -178,17 +178,17 @@ config KASAN_TAGS_IDENTIFYmemoryconsumption.configKASAN_VMALLOC-bool"Back mappings in vmalloc space with real shadow memory"-depends onKASAN_GENERIC&&HAVE_ARCH_KASAN_VMALLOC+bool"Check accesses to vmalloc allocations"+depends onHAVE_ARCH_KASAN_VMALLOChelp-Bydefault,theshadowregionforvmallocspaceistheread-only-zeropage.ThismeansthatKASANcannotdetecterrorsinvolving-vmallocspace.--Enablingthisoptionwillhookintovmap/vmallocandbackthose-mappingswithrealshadowmemoryallocatedondemand.Thisallows-forKASANtodetectmoresortsoferrors(andtosupportvmapped-stacks),butatthecostofhighermemoryusage.+ThismodemakesKASANcheckaccessestovmallocallocationsfor+validity.++WithsoftwareKASANmodes,checkingisdoneforalltypesofvmalloc+allocations.Enablingthisoptionleadstohighermemoryusage.++Withhardwaretag-basedKASAN,onlyVM_ALLOCmappingsarechecked.+Thereisnoadditionalmemoryusage.configKASAN_KUNIT_TESTtristate"KUnit-compatible tests of KASAN bug detection capabilities"if!KUNIT_ALL_TESTS
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Update KASAN documentation:
- Bump Clang version requirement for HW_TAGS as ARM64_MTE depends on
AS_HAS_LSE_ATOMICS as of commit 2decad92f4731 ("arm64: mte: Ensure
TIF_MTE_ASYNC_FAULT is set atomically"), which requires Clang 12.
- Add description of the new kasan.vmalloc command line flag.
- Mention that SW_TAGS and HW_TAGS modes now support vmalloc tagging.
- Explicitly say that the "Shadow memory" section is only applicable
to software KASAN modes.
- Mention that shadow-based KASAN_VMALLOC is supported on arm64.
Signed-off-by: Andrey Konovalov <redacted>
---
Documentation/dev-tools/kasan.rst | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
@@ -30,7 +30,7 @@ Software tag-based KASAN mode is only supported in Clang. The hardware KASAN mode (#3) relies on hardware to perform the checks but still requires a compiler version that supports memory tagging instructions.-This mode is supported in GCC 10+ and Clang 11+.+This mode is supported in GCC 10+ and Clang 12+. Both software KASAN modes work with SLUB and SLAB memory allocators, while the hardware tag-based KASAN currently only supports SLUB.
@@ -206,6 +206,9 @@ additional boot parameters that allow disabling KASAN or controlling features: Asymmetric mode: a bad access is detected synchronously on reads and asynchronously on writes.+-``kasan.vmalloc=off`` or ``=on`` disables or enables tagging of vmalloc+ allocations (default: ``on``).+-``kasan.stacktrace=off`` or ``=on`` disables or enables alloc and free stack traces collection (default: ``on``).
@@ -279,8 +282,8 @@ Software tag-based KASAN uses 0xFF as a match-all pointer tag (accesses through pointers with the 0xFF pointer tag are not checked). The value 0xFE is currently reserved to tag freed memory regions.-Software tag-based KASAN currently only supports tagging of slab and page_alloc-memory.+Software tag-based KASAN currently only supports tagging of slab, page_alloc,+and vmalloc memory. Hardware tag-based KASAN ~~~~~~~~~~~~~~~~~~~~~~~~
@@ -303,8 +306,8 @@ Hardware tag-based KASAN uses 0xFF as a match-all pointer tag (accesses through pointers with the 0xFF pointer tag are not checked). The value 0xFE is currently reserved to tag freed memory regions.-Hardware tag-based KASAN currently only supports tagging of slab and page_alloc-memory.+Hardware tag-based KASAN currently only supports tagging of slab, page_alloc,+and VM_ALLOC-based vmalloc memory. If the hardware does not support MTE (pre ARMv8.5), hardware tag-based KASAN will not be enabled. In this case, all KASAN boot parameters are ignored.
@@ -319,6 +322,8 @@ checking gets disabled. Shadow memory -------------+The contents of this section are only applicable to software KASAN modes.+ The kernel maps memory in several different parts of the address space. The range of kernel virtual addresses is large: there is not enough real memory to support a real shadow region for every address that could be
@@ -349,7 +354,7 @@ CONFIG_KASAN_VMALLOC With ``CONFIG_KASAN_VMALLOC``, KASAN can cover vmalloc space at the cost of greater memory usage. Currently, this is supported on x86,-riscv, s390, and powerpc.+arm64, riscv, s390, and powerpc. This works by hooking into vmalloc and vmap and dynamically allocating real shadow memory to back the mappings.
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Andrey Konovalov <redacted>
Generic KASAN already selects KASAN_VMALLOC to allow VMAP_STACK to be
selected unconditionally, see commit acc3042d62cb9 ("arm64: Kconfig:
select KASAN_VMALLOC if KANSAN_GENERIC is enabled").
The same change is needed for SW_TAGS KASAN.
HW_TAGS KASAN does not require enabling KASAN_VMALLOC for VMAP_STACK,
they already work together as is. Still, selecting KASAN_VMALLOC still
makes sense to make vmalloc() always protected. In case any bugs in
KASAN's vmalloc() support are discovered, the command line kasan.vmalloc
flag can be used to disable vmalloc() checking.
Select KASAN_VMALLOC for all KASAN modes for arm64.
Signed-off-by: Andrey Konovalov <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
---
Changes v2->v3:
- Update patch description.
Changes v1->v2:
- Split out this patch.
---
arch/arm64/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Andrey Konovalov <redacted>
Update the existing vmalloc_oob() test to account for the specifics
of the tag-based modes. Also add a few new checks and comments.
Add new vmalloc-related tests:
- vmalloc_helpers_tags() to check that exported vmalloc helpers can
handle tagged pointers.
- vmap_tags() to check that SW_TAGS mode properly tags vmap() mappings.
- vm_map_ram_tags() to check that SW_TAGS mode properly tags
vm_map_ram() mappings.
- vmalloc_percpu() to check that SW_TAGS mode tags regions allocated
for __alloc_percpu(). The tagging of per-cpu mappings is best-effort;
proper tagging is tracked in [1].
[1] https://bugzilla.kernel.org/show_bug.cgi?id=215019
Signed-off-by: Andrey Konovalov <redacted>
---
lib/test_kasan.c | 189 +++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 183 insertions(+), 6 deletions(-)
@@ -1049,21 +1050,181 @@ static void kmalloc_double_kzfree(struct kunit *test)KUNIT_EXPECT_KASAN_FAIL(test,kfree_sensitive(ptr));}+staticvoidvmalloc_helpers_tags(structkunit*test)+{+void*ptr;+intrv;++/* This test is intended for tag-based modes. */+KASAN_TEST_NEEDS_CONFIG_OFF(test,CONFIG_KASAN_GENERIC);++KASAN_TEST_NEEDS_CONFIG_ON(test,CONFIG_KASAN_VMALLOC);++ptr=vmalloc(PAGE_SIZE);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,ptr);++/* Check that the returned pointer is tagged. */+KUNIT_EXPECT_GE(test,(u8)get_tag(ptr),(u8)KASAN_TAG_MIN);+KUNIT_EXPECT_LT(test,(u8)get_tag(ptr),(u8)KASAN_TAG_KERNEL);++/* Make sure exported vmalloc helpers handle tagged pointers. */+KUNIT_ASSERT_TRUE(test,is_vmalloc_addr(ptr));+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,vmalloc_to_page(ptr));++/* Make sure vmalloc'ed memory permissions can be changed. */+rv=set_memory_ro((unsignedlong)ptr,1);+KUNIT_ASSERT_GE(test,rv,0);+rv=set_memory_rw((unsignedlong)ptr,1);+KUNIT_ASSERT_GE(test,rv,0);++vfree(ptr);+}+staticvoidvmalloc_oob(structkunit*test){-void*area;+char*v_ptr,*p_ptr;+structpage*page;+size_tsize=PAGE_SIZE/2-KASAN_GRANULE_SIZE-5;KASAN_TEST_NEEDS_CONFIG_ON(test,CONFIG_KASAN_VMALLOC);+v_ptr=vmalloc(size);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,v_ptr);+/*-*Wehavetobecarefulnottohittheguardpage.+*Wehavetobecarefulnottohittheguardpageinvmalloctests.*TheMMUwillcatchthatandcrashus.*/-area=vmalloc(3000);-KUNIT_ASSERT_NOT_ERR_OR_NULL(test,area);-KUNIT_EXPECT_KASAN_FAIL(test,((volatilechar*)area)[3100]);-vfree(area);+/* Make sure in-bounds accesses are valid. */+v_ptr[0]=0;+v_ptr[size-1]=0;++/*+*Anunalignedaccesspasttherequestedvmallocsize.+*OnlygenericKASANcanpreciselydetectthese.+*/+if(IS_ENABLED(CONFIG_KASAN_GENERIC))+KUNIT_EXPECT_KASAN_FAIL(test,((volatilechar*)v_ptr)[size]);++/* An aligned access into the first out-of-bounds granule. */+KUNIT_EXPECT_KASAN_FAIL(test,((volatilechar*)v_ptr)[size+5]);++/* Check that in-bounds accesses to the physical page are valid. */+page=vmalloc_to_page(v_ptr);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,page);+p_ptr=page_address(page);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,p_ptr);+p_ptr[0]=0;++vfree(v_ptr);++/*+*Wecan'tcheckforuse-after-unmapbugsinthisnorinthefollowing+*vmalloctests,asthepagemightbefullyunmappedandaccessingit+*willcrashthekernel.+*/+}++staticvoidvmap_tags(structkunit*test)+{+char*p_ptr,*v_ptr;+structpage*p_page,*v_page;+size_torder=1;++/*+*Thistestisspecificallycraftedforthesoftwaretag-basedmode,+*theonlytag-basedmodethatpoisonsvmapmappings.+*/+KASAN_TEST_NEEDS_CONFIG_ON(test,CONFIG_KASAN_SW_TAGS);++KASAN_TEST_NEEDS_CONFIG_ON(test,CONFIG_KASAN_VMALLOC);++p_page=alloc_pages(GFP_KERNEL,order);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,p_page);+p_ptr=page_address(p_page);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,p_ptr);++v_ptr=vmap(&p_page,1<<order,VM_MAP,PAGE_KERNEL);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,v_ptr);++/*+*Wecan'tcheckforout-of-boundsbugsinthisnorinthefollowing+*vmalloctests,asallocationshavepagegranularityandaccessing+*theguardpagewillcrashthekernel.+*/++KUNIT_EXPECT_GE(test,(u8)get_tag(v_ptr),(u8)KASAN_TAG_MIN);+KUNIT_EXPECT_LT(test,(u8)get_tag(v_ptr),(u8)KASAN_TAG_KERNEL);++/* Make sure that in-bounds accesses through both pointers work. */+*p_ptr=0;+*v_ptr=0;++/* Make sure vmalloc_to_page() correctly recovers the page pointer. */+v_page=vmalloc_to_page(v_ptr);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,v_page);+KUNIT_EXPECT_PTR_EQ(test,p_page,v_page);++vunmap(v_ptr);+free_pages((unsignedlong)p_ptr,order);+}++staticvoidvm_map_ram_tags(structkunit*test)+{+char*p_ptr,*v_ptr;+structpage*page;+size_torder=1;++/*+*Thistestisspecificallycraftedforthesoftwaretag-basedmode,+*theonlytag-basedmodethatpoisonsvm_map_rammappings.+*/+KASAN_TEST_NEEDS_CONFIG_ON(test,CONFIG_KASAN_SW_TAGS);++page=alloc_pages(GFP_KERNEL,order);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,page);+p_ptr=page_address(page);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,p_ptr);++v_ptr=vm_map_ram(&page,1<<order,-1);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,v_ptr);++KUNIT_EXPECT_GE(test,(u8)get_tag(v_ptr),(u8)KASAN_TAG_MIN);+KUNIT_EXPECT_LT(test,(u8)get_tag(v_ptr),(u8)KASAN_TAG_KERNEL);++/* Make sure that in-bounds accesses through both pointers work. */+*p_ptr=0;+*v_ptr=0;++vm_unmap_ram(v_ptr,1<<order);+free_pages((unsignedlong)p_ptr,order);+}++staticvoidvmalloc_percpu(structkunit*test)+{+char__percpu*ptr;+intcpu;++/*+*Thistestisspecificallycraftedforthesoftwaretag-basedmode,+*theonlytag-basedmodethatpoisonspercpumappings.+*/+KASAN_TEST_NEEDS_CONFIG_ON(test,CONFIG_KASAN_SW_TAGS);++ptr=__alloc_percpu(PAGE_SIZE,PAGE_SIZE);++for_each_possible_cpu(cpu){+char*c_ptr=per_cpu_ptr(ptr,cpu);++KUNIT_EXPECT_GE(test,(u8)get_tag(c_ptr),(u8)KASAN_TAG_MIN);+KUNIT_EXPECT_LT(test,(u8)get_tag(c_ptr),(u8)KASAN_TAG_KERNEL);++/* Make sure that in-bounds accesses don't crash the kernel. */+*c_ptr=0;+}++free_percpu(ptr);}/*
@@ -1097,6 +1258,18 @@ static void match_all_not_assigned(struct kunit *test)KUNIT_EXPECT_LT(test,(u8)get_tag(ptr),(u8)KASAN_TAG_KERNEL);free_pages((unsignedlong)ptr,order);}++if(!IS_ENABLED(CONFIG_KASAN_VMALLOC))+return;++for(i=0;i<256;i++){+size=(get_random_int()%1024)+1;+ptr=vmalloc(size);+KUNIT_ASSERT_NOT_ERR_OR_NULL(test,ptr);+KUNIT_EXPECT_GE(test,(u8)get_tag(ptr),(u8)KASAN_TAG_MIN);+KUNIT_EXPECT_LT(test,(u8)get_tag(ptr),(u8)KASAN_TAG_KERNEL);+vfree(ptr);+}}/* Check that 0xff works as a match-all pointer tag for tag-based modes. */
On Thu, Dec 30, 2021 at 8:12 PM [off-list ref] wrote:
From: Andrey Konovalov <redacted>
Hi,
This patchset adds vmalloc tagging support for SW_TAGS and HW_TAGS
KASAN modes.
The tree with patches is available here:
https://github.com/xairy/linux/tree/up-kasan-vmalloc-tags-v5-akpm
About half of patches are cleanups I went for along the way. None of
them seem to be important enough to go through stable, so I decided
not to split them out into separate patches/series.
The patchset is partially based on an early version of the HW_TAGS
patchset by Vincenzo that had vmalloc support. Thus, I added a
Co-developed-by tag into a few patches.
SW_TAGS vmalloc tagging support is straightforward. It reuses all of
the generic KASAN machinery, but uses shadow memory to store tags
instead of magic values. Naturally, vmalloc tagging requires adding
a few kasan_reset_tag() annotations to the vmalloc code.
HW_TAGS vmalloc tagging support stands out. HW_TAGS KASAN is based on
Arm MTE, which can only assigns tags to physical memory. As a result,
HW_TAGS KASAN only tags vmalloc() allocations, which are backed by
page_alloc memory. It ignores vmap() and others.
Thanks!
Hi Andrew,
Could you PTAL and consider taking this into mm?
Thanks!
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Kuan-Ying Lee <hidden> Date: 2022-01-03 02:33:04
On Fri, 2021-12-31 at 03:14 +0800, andrey.konovalov@linux.dev wrote:
quoted hunk
From: Andrey Konovalov <redacted>
Add a new GFP flag __GFP_SKIP_ZERO that allows to skip memory
initialization. The flag is only effective with HW_TAGS KASAN.
This flag will be used by vmalloc code for page_alloc allocations
backing vmalloc() mappings in a following patch. The reason to skip
memory initialization for these pages in page_alloc is because
vmalloc
code will be initializing them instead.
With the current implementation, when __GFP_SKIP_ZERO is provided,
__GFP_ZEROTAGS is ignored. This doesn't matter, as these two flags
are
never provided at the same time. However, if this is changed in the
future, this particular implementation detail can be changed as well.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v4->v5:
- Cosmetic changes to __def_gfpflag_names_kasan and __GFP_BITS_SHIFT.
Changes v3->v4:
- Only define __GFP_SKIP_ZERO when CONFIG_KASAN_HW_TAGS is enabled.
- Add __GFP_SKIP_ZERO to include/trace/events/mmflags.h.
- Use proper kasan_hw_tags_enabled() check instead of
IS_ENABLED(CONFIG_KASAN_HW_TAGS). Also add explicit checks for
software modes.
Changes v2->v3:
- Update patch description.
Changes v1->v2:
- Add this patch.
---
include/linux/gfp.h | 18 +++++++++++-------
include/trace/events/mmflags.h | 1 +
mm/page_alloc.c | 18 +++++++++++++++++-
3 files changed, 29 insertions(+), 8 deletions(-)
memory itself
- * is being zeroed (either via __GFP_ZERO or via init_on_alloc).
This flag is
- * intended for optimization: setting memory tags at the same time
as zeroing
- * memory has minimal additional performace impact.
+ * is being zeroed (either via __GFP_ZERO or via init_on_alloc,
provided that
+ * __GFP_SKIP_ZERO is not set). This flag is intended for
optimization: setting
+ * memory tags at the same time as zeroing memory has minimal
additional
+ * performace impact.
*
* %__GFP_SKIP_KASAN_UNPOISON makes KASAN skip unpoisoning on page
allocation.
* Only effective in HW_TAGS mode.
On Mon, Jan 3, 2022 at 3:32 AM Kuan-Ying Lee [off-list ref] wrote:
On Fri, 2021-12-31 at 03:14 +0800, andrey.konovalov@linux.dev wrote:
quoted
From: Andrey Konovalov <redacted>
Add a new GFP flag __GFP_SKIP_ZERO that allows to skip memory
initialization. The flag is only effective with HW_TAGS KASAN.
This flag will be used by vmalloc code for page_alloc allocations
backing vmalloc() mappings in a following patch. The reason to skip
memory initialization for these pages in page_alloc is because
vmalloc
code will be initializing them instead.
With the current implementation, when __GFP_SKIP_ZERO is provided,
__GFP_ZEROTAGS is ignored. This doesn't matter, as these two flags
are
never provided at the same time. However, if this is changed in the
future, this particular implementation detail can be changed as well.
Signed-off-by: Andrey Konovalov <redacted>
---
Changes v4->v5:
- Cosmetic changes to __def_gfpflag_names_kasan and __GFP_BITS_SHIFT.
Changes v3->v4:
- Only define __GFP_SKIP_ZERO when CONFIG_KASAN_HW_TAGS is enabled.
- Add __GFP_SKIP_ZERO to include/trace/events/mmflags.h.
- Use proper kasan_hw_tags_enabled() check instead of
IS_ENABLED(CONFIG_KASAN_HW_TAGS). Also add explicit checks for
software modes.
Changes v2->v3:
- Update patch description.
Changes v1->v2:
- Add this patch.
---
include/linux/gfp.h | 18 +++++++++++-------
include/trace/events/mmflags.h | 1 +
mm/page_alloc.c | 18 +++++++++++++++++-
3 files changed, 29 insertions(+), 8 deletions(-)
memory itself
- * is being zeroed (either via __GFP_ZERO or via init_on_alloc).
This flag is
- * intended for optimization: setting memory tags at the same time
as zeroing
- * memory has minimal additional performace impact.
+ * is being zeroed (either via __GFP_ZERO or via init_on_alloc,
provided that
+ * __GFP_SKIP_ZERO is not set). This flag is intended for
optimization: setting
+ * memory tags at the same time as zeroing memory has minimal
additional
+ * performace impact.
*
* %__GFP_SKIP_KASAN_UNPOISON makes KASAN skip unpoisoning on page
allocation.
* Only effective in HW_TAGS mode.
should_skip_kasan_unpoison(gfp_t flags, bool init_tags)
return init_tags || (flags & __GFP_SKIP_KASAN_UNPOISON);
}
+static inline bool should_skip_init(gfp_t flags)
+{
+ /* Don't skip if a software KASAN mode is enabled. */
+ if (IS_ENABLED(CONFIG_KASAN_GENERIC) ||
+ IS_ENABLED(CONFIG_KASAN_SW_TAGS))
+ return false;
Forget to drop the above check?
I saw v4 mentioned that this check can be dropped. [1]
Do I miss something?
[1] https://lkml.org/lkml/2021/12/30/450
Right, forgot to include this change. Will include into v5 or post as
a standalone fix closer to rc1. Thanks for noticing!
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel