[PATCH bpf] bpf: check cpu_possible() in per-cpu helpers

Three paths take a target CPU id as a raw u32 and only reject
cpu >= nr_cpu_ids before the value is used as a per_cpu_ptr() index.

bpf_map_lookup_percpu_elem() dispatches to the per-cpu map
implementations, and all three of them (percpu array, percpu hash,
percpu LRU hash) check nothing beyond that upper bound:

	if (cpu >= nr_cpu_ids)
		return NULL;
	...
	return per_cpu_ptr(..., cpu);

bpf_per_cpu_ptr() has the same weak check before returning
per_cpu_ptr(ptr, cpu) for a kernel percpu variable.

Note that the verifier rewrites calls to bpf_map_lookup_percpu_elem()
with a known map into direct calls to the map callback
(kernel/bpf/fixups.c, BPF_FUNC_map_lookup_percpu_elem case), so a
check in the helpers.c wrapper alone does not cover JITed programs;
the callbacks themselves must reject the hole CPU id.

Per-cpu memory is only allocated for possible CPUs, and
__per_cpu_offset[] is only initialized for them (on arm64,
setup_per_cpu_areas() walks for_each_possible_cpu).  On a system
with a sparse possible mask - e.g. an arm64 guest with a device-tree
CPU hole, possible mask 0,2-3 with nr_cpu_ids = 4 - a lookup with
the hole CPU id passes the weak checks, and per_cpu_ptr() computes
the returned pointer from the uninitialized __per_cpu_offset[hole].

For bpf_map_lookup_percpu_elem() the returned pointer lies outside
any valid per-cpu chunk of the map: reads leak the stale bytes
through the map value, and writes corrupt whatever the offset
happens to land on.  For bpf_per_cpu_ptr() the leak target is kernel
memory instead of map values (the return type is marked MEM_RDONLY,
so reads only).

Reproduced on an arm64 guest with possible mask 0,2-3 (Ubuntu
7.3.0-9-generic): a socket filter program passes the pointer
returned by bpf_map_lookup_percpu_elem(map, key, 1) on as the value
argument of bpf_map_update_elem(), and the copy inside
array_map_update_elem() faults on it:

  pc : __pi_memcpy_generic+0x3c/0x230
  lr : array_map_update_elem+0x148/0x2a8
  Call trace:
   __pi_memcpy_generic+0x3c/0x230 (P)
   bpf_prog_835a4ba2ea6755f0_poc+0x90/0xc0
   bpf_test_run+0x2d0/0x460
   bpf_prog_test_run_skb+0x4ac/0x920
   __sys_bpf+0x614/0xbe8
   __arm64_sys_bpf+0x38/0x78

the memcpy source (x1 = 0000b7737ec7dc88) being the out-of-bounds
pointer the lookup returned for the hole CPU.

The syscall side of the same problem (BPF_F_CPU map operation flags)
was fixed by ed54bf564ac5 ("bpf: Fix BPF_F_CPU validation for sparse
CPU IDs"), which added cpu_possible() validation to
bpf_map_check_op_flags().  These paths do not go through that
function, so the same class of hole CPU id needs to be rejected here
as well.

Add the check to the three map callbacks (so the verifier's direct
call path is covered), keep it in the helper dispatch as a central
fallback, and fix bpf_per_cpu_ptr() in place.

Suggested-by: sashiko-bot <sashiko-bot@kernel.org>
Fixes: 07343110b293 ("bpf: add bpf_map_lookup_percpu_elem for percpu map")
Fixes: eaa6bcb71ef6 ("bpf: Introduce bpf_per_cpu_ptr()")
Cc: <redacted>
Signed-off-by: 平之 <redacted>
---
Changes in v2: add the cpu_possible() check to the three per-cpu map
callbacks as well, because the verifier rewrites helper calls with a
known map into direct calls to the callbacks (kernel/bpf/fixups.c)
and thus bypasses the helpers.c wrapper, as pointed out by
sashiko-bot.

Link: https://lore.kernel.org/bpf/sashiko-outbox-186048@kernel.org/ (local)
---
 kernel/bpf/arraymap.c | 2 +-
 kernel/bpf/hashtab.c  | 4 ++--
 kernel/bpf/helpers.c  | 6 +++++-
 3 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 133f189bf6d2..eab03f985280 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c

@@ -327,7 +327,7 @@ static void *percpu_array_map_lookup_percpu_elem(struct bpf_map *map, void *key,
 	struct bpf_array *array = container_of(map, struct bpf_array, map);
 	u32 index = *(u32 *)key;
 
-	if (cpu >= nr_cpu_ids)
+	if (cpu >= nr_cpu_ids || !cpu_possible(cpu))
 		return NULL;
 
 	if (unlikely(index >= array->map.max_entries))

diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index bf78e09f4d51..007279c16b77 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c

@@ -2670,7 +2670,7 @@ static void *htab_percpu_map_lookup_percpu_elem(struct bpf_map *map, void *key,
 {
 	struct htab_elem *l;
 
-	if (cpu >= nr_cpu_ids)
+	if (cpu >= nr_cpu_ids || !cpu_possible(cpu))
 		return NULL;
 
 	l = __htab_map_lookup_elem(map, key);

@@ -2696,7 +2696,7 @@ static void *htab_lru_percpu_map_lookup_percpu_elem(struct bpf_map *map, void *k
 {
 	struct htab_elem *l;
 
-	if (cpu >= nr_cpu_ids)
+	if (cpu >= nr_cpu_ids || !cpu_possible(cpu))
 		return NULL;
 
 	l = __htab_map_lookup_elem(map, key);

diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 648abe1ba63a..f9f2bd665038 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c

@@ -133,6 +133,10 @@ const struct bpf_func_proto bpf_map_peek_elem_proto = {
 BPF_CALL_3(bpf_map_lookup_percpu_elem, struct bpf_map *, map, void *, key, u32, cpu)
 {
 	WARN_ON_ONCE(!bpf_rcu_lock_held());
+
+	if (cpu >= nr_cpu_ids || !cpu_possible(cpu))
+		return 0;
+
 	return (unsigned long) map->ops->map_lookup_percpu_elem(map, key, cpu);
 }
 

@@ -753,7 +757,7 @@ const struct bpf_func_proto bpf_copy_from_user_task_proto = {
 
 BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
 {
-	if (cpu >= nr_cpu_ids)
+	if (cpu >= nr_cpu_ids || !cpu_possible(cpu))
 		return (unsigned long)NULL;
 
 	return (unsigned long)per_cpu_ptr((const void __percpu *)(const uintptr_t)ptr, cpu);

-- 
2.39.5 (Apple Git-154)