This patch initializes KASan shadow region's page table and memory.
There are two stage for KASan initializing:

1. At early boot stage the whole shadow region is mapped to just
   one physical page (kasan_zero_page). It is finished by the function
   kasan_early_init which is called by __mmap_switched(arch/arm/kernel/
   head-common.S)

2. After the calling of paging_init, we use kasan_zero_page as zero
   shadow for some memory that KASan does not need to track, and we
   allocate a new shadow space for the other memory that KASan need to
   track. These issues are finished by the function kasan_init which is
   call by setup_arch.

When using KASan we also need to increase the THREAD_SIZE_ORDER
from 1 to 2 as the extra calls for shadow memory uses quite a bit
of stack.

As we need to make a temporary copy of the PGD when setting up
shadow memory we create a helpful PGD_SIZE definition for both
LPAE and non-LPAE setups.

The KASan core code unconditionally calls pud_populate() so this
needs to be changed from BUG() to do {} while (0) when building
with KASan enabled.

After the initial development by Andre Ryabinin several modifications
have been made to this code:

Abbott Liu [off-list ref]
- Add support ARM LPAE: If LPAE is enabled, KASan shadow region's
  mapping table need be copied in the pgd_alloc() function.
- Change kasan_pte_populate,kasan_pmd_populate,kasan_pud_populate,
  kasan_pgd_populate from .meminit.text section to .init.text section.
  Reported by Florian Fainelli [off-list ref]

Linus Walleij [off-list ref]:
- Drop the custom mainpulation of TTBR0 and just use
  cpu_switch_mm() to switch the pgd table.
- Adopt to handle 4th level page tabel folding.
- Rewrite the entire page directory and page entry initialization
  sequence to be recursive based on ARM64:s kasan_init.c.

Ard Biesheuvel [off-list ref]:
- Necessary underlying fixes.
- Crucial bug fixes to the memory set-up code.

Cc: Alexander Potapenko <glider@google.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: kasan-dev@googlegroups.com
Cc: Mike Rapoport <redacted>
Co-developed-by: Andrey Ryabinin <redacted>
Co-developed-by: Abbott Liu <redacted>
Co-developed-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Mike Rapoport <redacted>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Tested-by: Ard Biesheuvel <ardb@kernel.org> # QEMU/KVM/mach-virt/LPAE/8G
Reported-by: Russell King - ARM Linux <linux@armlinux.org.uk>
Reported-by: Florian Fainelli <f.fainelli@gmail.com>
Signed-off-by: Andrey Ryabinin <redacted>
Signed-off-by: Abbott Liu <redacted>
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Linus Walleij <redacted>
---
ChangeLog v14->v15:
- Avoids reallocating KASAN blocks when a range gets
  mapped twice - this occurs when mapping the DTB space explicitly.
- Insert a missing TLB flush.
- Move the cache flush after switching the MM (which makes logical
  sense.
- All these fixes discovered by Ard Bisheuvel.
- Dropped the special mapping around the DTB after using Ard's
  patches for remapping the DTB in a special memory area.
- Add asmlinkage prototype for kasan_early_init() to get
  rid of some compilation warnings.
ChangeLog v13->v14:
- Provide more elaborate prints of how virtual kernel memory
  is mapped to the allocated lowmem pages.
- Make sure to also map the memory around the __atags_pointer:
  this memory is used for the device tree blob (DTB) and will be
  accessed by the device tree parser. We were just lucky that
  this was mostly in some acceptable memory location until now.
ChangeLog v12->v13:
- Rebase on kernel v5.9-rc1
ChangeLog v11->v12:
- Do not try to shadow highmem memory blocks. (Ard)
- Provoke a build bug if the entire shadow memory doesn't fit
  inside a single pgd_index() (Ard)
- Move the pointer to (unsigned long) casts into the create_mapping()
  function. (Ard)
- After setting up the shadow memory make sure to issue
  local_flush_tlb_all() so that we refresh all the global mappings. (Ard)
- Simplify pte_populate() (Ard)
- Skip over pud population as well as p4d. (Ard)
- Drop the stop condition pmd_none(*pmdp) in the pmd population
  loop. (Ard)
- Stop passing around the node (NUMA) parameter in the init code,
  we are not expecting any NUMA architectures to be introduced into
  ARM32 so just hardcode NUMA_NO_NODE when calling
  memblock_alloc_try_nid().
ChangeLog v10->v11:
- Fix compilation on LPAE systems.
- Move the check for valid pgdp, pudp and pmdp into the loop for
  each level moving over the directory pointers: we were just lucky
  that we just needed one directory for each level so this fixes
  the pmdp issue with LPAE and KASan now works like a charm on
  LPAE as well.
- Fold fourth level page directory (p4d) into the global page directory
  pgd and just skip into the page upper directory (pud) directly. We
  do not anticipate that ARM32 will every use 5-level page tables.
- Simplify the ifdeffery around the temporary pgd.
- Insert a comment about pud_populate() that is unconditionally called
  by the KASan core code.
ChangeLog v9->v10:
- Rebase onto v5.8-rc1
- add support for folded p4d page tables, use the primitives necessary
  for the 4th level folding, add (empty) walks of p4d level.
- Use the <linux/pgtable.h> header file that has now appeared as part
  of the VM consolidation series.
- Use a recursive method to walk pgd/p4d/pud/pmd/pte instead of the
  separate early/main calls and the flat call structure used in the
  old code. This was inspired by the ARM64 KASan init code.
- Assume authorship of this code, I have now written the majority of
  it so the blame is on me and noone else.
ChangeLog v8->v9:
- Drop the custom CP15 manipulation and cache flushing for swapping
  TTBR0 and instead just use cpu_switch_mm().
- Collect Ard's tags.
ChangeLog v7->v8:
- Rebased.
ChangeLog v6->v7:
- Use SPDX identifer for the license.
- Move the TTBR0 accessor calls into this patch.
---
 arch/arm/include/asm/kasan.h       |  33 ++++
 arch/arm/include/asm/pgalloc.h     |   8 +-
 arch/arm/include/asm/thread_info.h |   8 +
 arch/arm/kernel/head-common.S      |   3 +
 arch/arm/kernel/setup.c            |   2 +
 arch/arm/mm/Makefile               |   3 +
 arch/arm/mm/kasan_init.c           | 284 +++++++++++++++++++++++++++++
 arch/arm/mm/pgd.c                  |  16 +-
 8 files changed, 355 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm/include/asm/kasan.h
 create mode 100644 arch/arm/mm/kasan_init.c

diff --git a/arch/arm/include/asm/kasan.h b/arch/arm/include/asm/kasan.h
new file mode 100644
index 000000000000..303c35df3135
--- /dev/null
+++ b/arch/arm/include/asm/kasan.h

@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * arch/arm/include/asm/kasan.h
+ *
+ * Copyright (c) 2015 Samsung Electronics Co., Ltd.
+ * Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
+ *
+ */
+
+#ifndef __ASM_KASAN_H
+#define __ASM_KASAN_H
+
+#ifdef CONFIG_KASAN
+
+#include <asm/kasan_def.h>
+
+#define KASAN_SHADOW_SCALE_SHIFT 3
+
+/*
+ * The compiler uses a shadow offset assuming that addresses start
+ * from 0. Kernel addresses don't start from 0, so shadow
+ * for kernel really starts from 'compiler's shadow offset' +
+ * ('kernel address space start' >> KASAN_SHADOW_SCALE_SHIFT)
+ */
+
+asmlinkage void kasan_early_init(void);
+extern void kasan_init(void);
+
+#else
+static inline void kasan_init(void) { }
+#endif
+
+#endif

diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h
index 15f4674715f8..fdee1f04f4f3 100644
--- a/arch/arm/include/asm/pgalloc.h
+++ b/arch/arm/include/asm/pgalloc.h

@@ -21,6 +21,7 @@
 #define _PAGE_KERNEL_TABLE     (PMD_TYPE_TABLE | PMD_BIT4 | PMD_DOMAIN(DOMAIN_KERNEL))

 #ifdef CONFIG_ARM_LPAE
+#define PGD_SIZE               (PTRS_PER_PGD * sizeof(pgd_t))

 static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
 {

@@ -28,14 +29,19 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
 }

 #else  /* !CONFIG_ARM_LPAE */
+#define PGD_SIZE               (PAGE_SIZE << 2)

 /*
  * Since we have only two-level page tables, these are trivial
  */
 #define pmd_alloc_one(mm,addr)         ({ BUG(); ((pmd_t *)2); })
 #define pmd_free(mm, pmd)              do { } while (0)
+#ifdef CONFIG_KASAN
+/* The KASan core unconditionally calls pud_populate() on all architectures */
+#define pud_populate(mm,pmd,pte)       do { } while (0)
+#else
 #define pud_populate(mm,pmd,pte)       BUG()
-
+#endif
 #endif /* CONFIG_ARM_LPAE */

 extern pgd_t *pgd_alloc(struct mm_struct *mm);

diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h
index 536b6b979f63..56fae7861fd3 100644
--- a/arch/arm/include/asm/thread_info.h
+++ b/arch/arm/include/asm/thread_info.h

@@ -13,7 +13,15 @@
 #include <asm/fpstate.h>
 #include <asm/page.h>

+#ifdef CONFIG_KASAN
+/*
+ * KASan uses a lot of extra stack space so the thread size order needs to
+ * be increased.
+ */
+#define THREAD_SIZE_ORDER      2
+#else
 #define THREAD_SIZE_ORDER      1
+#endif
 #define THREAD_SIZE            (PAGE_SIZE << THREAD_SIZE_ORDER)
 #define THREAD_START_SP                (THREAD_SIZE - 8)

diff --git a/arch/arm/kernel/head-common.S b/arch/arm/kernel/head-common.S
index 6840c7c60a85..89c80154b9ef 100644
--- a/arch/arm/kernel/head-common.S
+++ b/arch/arm/kernel/head-common.S

@@ -111,6 +111,9 @@ __mmap_switched:
        str     r8, [r2]                        @ Save atags pointer
        cmp     r3, #0
        strne   r10, [r3]                       @ Save control register values
+#ifdef CONFIG_KASAN
+       bl      kasan_early_init
+#endif
        mov     lr, #0
        b       start_kernel
 ENDPROC(__mmap_switched)

diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
index 2a70e4958c14..43d033696e33 100644
--- a/arch/arm/kernel/setup.c
+++ b/arch/arm/kernel/setup.c

@@ -59,6 +59,7 @@
 #include <asm/unwind.h>
 #include <asm/memblock.h>
 #include <asm/virt.h>
+#include <asm/kasan.h>

 #include "atags.h"

@@ -1139,6 +1140,7 @@ void __init setup_arch(char **cmdline_p)
        early_ioremap_reset();

        paging_init(mdesc);
+       kasan_init();
        request_standard_resources(mdesc);

        if (mdesc->restart)

diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile
index 99699c32d8a5..4536159bc8fa 100644
--- a/arch/arm/mm/Makefile
+++ b/arch/arm/mm/Makefile

@@ -113,3 +113,6 @@ obj-$(CONFIG_CACHE_L2X0_PMU)        += cache-l2x0-pmu.o
 obj-$(CONFIG_CACHE_XSC3L2)     += cache-xsc3l2.o
 obj-$(CONFIG_CACHE_TAUROS2)    += cache-tauros2.o
 obj-$(CONFIG_CACHE_UNIPHIER)   += cache-uniphier.o
+
+KASAN_SANITIZE_kasan_init.o    := n
+obj-$(CONFIG_KASAN)            += kasan_init.o

diff --git a/arch/arm/mm/kasan_init.c b/arch/arm/mm/kasan_init.c
new file mode 100644
index 000000000000..22ac84defa5d
--- /dev/null
+++ b/arch/arm/mm/kasan_init.c

@@ -0,0 +1,284 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * This file contains kasan initialization code for ARM.
+ *
+ * Copyright (c) 2018 Samsung Electronics Co., Ltd.
+ * Author: Andrey Ryabinin <ryabinin.a.a@gmail.com>
+ * Author: Linus Walleij <linus.walleij@linaro.org>
+ */
+
+#define pr_fmt(fmt) "kasan: " fmt
+#include <linux/kasan.h>
+#include <linux/kernel.h>
+#include <linux/memblock.h>
+#include <linux/sched/task.h>
+#include <linux/start_kernel.h>
+#include <linux/pgtable.h>
+#include <asm/cputype.h>
+#include <asm/highmem.h>
+#include <asm/mach/map.h>
+#include <asm/memory.h>
+#include <asm/page.h>
+#include <asm/pgalloc.h>
+#include <asm/procinfo.h>
+#include <asm/proc-fns.h>
+
+#include "mm.h"
+
+static pgd_t tmp_pgd_table[PTRS_PER_PGD] __initdata __aligned(PGD_SIZE);
+
+pmd_t tmp_pmd_table[PTRS_PER_PMD] __page_aligned_bss;
+
+static __init void *kasan_alloc_block(size_t size)
+{
+       return memblock_alloc_try_nid(size, size, __pa(MAX_DMA_ADDRESS),
+                                     MEMBLOCK_ALLOC_KASAN, NUMA_NO_NODE);
+}
+
+static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr,
+                                     unsigned long end, bool early)
+{
+       unsigned long next;
+       pte_t *ptep = pte_offset_kernel(pmdp, addr);
+
+       do {
+               pte_t entry;
+               void *p;
+
+               next = addr + PAGE_SIZE;
+
+               if (!early) {
+                       if (!pte_none(READ_ONCE(*ptep)))
+                               continue;
+
+                       p = kasan_alloc_block(PAGE_SIZE);
+                       if (!p) {
+                               panic("%s failed to alloc pte for address 0x%lx\n",

+                       end = __va(arm_lowmem_limit);
+               }
+               if (start >= end) {
+                       pr_info("Skipping invalid memory block %px-%px\n",
+                               start, end);
+                       continue;
+               }
+
+               create_mapping(start, end);
+       }
+
+       /*
+        * 1. The module global variables are in MODULES_VADDR ~ MODULES_END,
+        *    so we need to map this area.
+        * 2. PKMAP_BASE ~ PKMAP_BASE+PMD_SIZE's shadow and MODULES_VADDR
+        *    ~ MODULES_END's shadow is in the same PMD_SIZE, so we can't
+        *    use kasan_populate_zero_shadow.
+        */
+       create_mapping((void *)MODULES_VADDR, (void *)(PKMAP_BASE + PMD_SIZE));
+
+       /*
+        * KAsan may reuse the contents of kasan_early_shadow_pte directly, so
+        * we should make sure that it maps the zero page read-only.
+        */
+       for (i = 0; i < PTRS_PER_PTE; i++)
+               set_pte_at(&init_mm, KASAN_SHADOW_START + i*PAGE_SIZE,
+                          &kasan_early_shadow_pte[i],
+                          pfn_pte(virt_to_pfn(kasan_early_shadow_page),
+                               __pgprot(pgprot_val(PAGE_KERNEL)
+                                        | L_PTE_RDONLY)));
+
+       cpu_switch_mm(swapper_pg_dir, &init_mm);
+       local_flush_tlb_all();
+
+       memset(kasan_early_shadow_page, 0, PAGE_SIZE);
+       pr_info("Kernel address sanitizer initialized\n");
+       init_task.kasan_depth = 0;
+}

diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c
index c5e1b27046a8..f8e9bc58a84f 100644
--- a/arch/arm/mm/pgd.c
+++ b/arch/arm/mm/pgd.c

@@ -66,7 +66,21 @@ pgd_t *pgd_alloc(struct mm_struct *mm)
        new_pmd = pmd_alloc(mm, new_pud, 0);
        if (!new_pmd)
                goto no_pmd;
-#endif
+#ifdef CONFIG_KASAN
+       /*
+        * Copy PMD table for KASAN shadow mappings.
+        */
+       init_pgd = pgd_offset_k(TASK_SIZE);
+       init_p4d = p4d_offset(init_pgd, TASK_SIZE);
+       init_pud = pud_offset(init_p4d, TASK_SIZE);
+       init_pmd = pmd_offset(init_pud, TASK_SIZE);
+       new_pmd = pmd_offset(new_pud, TASK_SIZE);
+       memcpy(new_pmd, init_pmd,
+              (pmd_index(MODULES_VADDR) - pmd_index(TASK_SIZE))
+              * sizeof(pmd_t));
+       clean_dcache_area(new_pmd, PTRS_PER_PMD * sizeof(pmd_t));
+#endif /* CONFIG_KASAN */
+#endif /* CONFIG_LPAE */

        if (!vectors_high()) {
                /*
--

2.26.2