[PATCH bpf-next 23/38] bpf: Check the maps that sleepable callbacks use
flat view
COOLING7d
SUPERSEDED
From: Alexei Starovoitov <hidden>
Date: 2026-10-17 01:29:58
Also in:
bpf
Subsystem:
bpf [core], bpf [general] (safe dynamic programs and tools), the rest · Maintainers:
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman, Kumar Kartikeya Dwivedi, Linus Torvalds
Revision v1 of 2 in this series.
Revisions (2)
- v1 current
- v2 [diff vs current]
From: Josef Bacik <josef@toxicpanda.com>
Sleepable programs can only use the kinds of map that do without an RCU
read side critical section. check_map_prog_compatibility() sees to that
when a map is bound to the program, and asks whether the program is
sleepable. That misses the bpf_wq or bpf_task_work callback of a program
that is not, which runs sleepable all the same, and it misses an inner
map, which is not bound.
The bpf_wq callback of a tc program that looks a key up in a sockhash
gets as far as
WARNING: net/core/sock_map.c:903 at __sock_hash_lookup_elem+0x18f/0x1e0
Ask the same question for the map that is handed to a helper in sleepable
context, unless the RCU read lock is held: by bpf_rcu_read_lock(), or by
the helper that runs the callback, as bpf_for_each_map_elem() does for a
hash map. Programs that take the lock go on working. Disabled preemption
or interrupts are not enough, since the maps ask for rcu_read_lock_held().
The other parts of the program may use such a map as before.
What is refused from now on runs into the warning above, or uses what the
map frees behind a plain RCU grace period: an entry of a devmap, of a perf
event array or of a cgroup array.
A bloom filter does without RCU. Add it to the list, so that callbacks
which use one go on loading. As a side effect sleepable programs can use
bloom filters as well.
A pointer to the value of a devmap which a program keeps beyond
bpf_rcu_read_unlock() is not dealt with here.
Fixes: 638e4b825d52 ("bpf: Allows per-cpu maps and map-in-map in sleepable programs")
Fixes: 8e83da9732d9 ("bpf: add bpf_wq_start")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/verifier.c | 80 +++++++++++++++++++++++++++----------------
1 file changed, 51 insertions(+), 29 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 1f2a846ae415..09173d1d66d3 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c@@ -219,6 +219,7 @@ static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg); static bool is_trusted_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg); static inline bool in_sleepable_context(struct bpf_verifier_env *env); +static bool in_rcu_callback(struct bpf_verifier_env *env); static const char *non_sleepable_context_description(struct bpf_verifier_env *env); static void scalar32_min_max_add(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg); static void scalar_min_max_add(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg);
@@ -10379,12 +10380,57 @@ bool bpf_allow_tail_call_in_subprogs(struct bpf_verifier_env *env) bpf_jit_supports_subprog_tailcalls(); } +/* May the map be used outside of an RCU read side critical section? */ +static bool map_needs_no_rcu(const struct bpf_map *map) +{ + switch (map->map_type) { + case BPF_MAP_TYPE_HASH: + case BPF_MAP_TYPE_RHASH: + case BPF_MAP_TYPE_LRU_HASH: + case BPF_MAP_TYPE_ARRAY: + case BPF_MAP_TYPE_PERCPU_HASH: + case BPF_MAP_TYPE_PERCPU_ARRAY: + case BPF_MAP_TYPE_LRU_PERCPU_HASH: + case BPF_MAP_TYPE_LPM_TRIE: + case BPF_MAP_TYPE_ARRAY_OF_MAPS: + case BPF_MAP_TYPE_HASH_OF_MAPS: + case BPF_MAP_TYPE_RINGBUF: + case BPF_MAP_TYPE_USER_RINGBUF: + case BPF_MAP_TYPE_INODE_STORAGE: + case BPF_MAP_TYPE_SK_STORAGE: + case BPF_MAP_TYPE_TASK_STORAGE: + case BPF_MAP_TYPE_CGRP_STORAGE: + case BPF_MAP_TYPE_QUEUE: + case BPF_MAP_TYPE_STACK: + case BPF_MAP_TYPE_ARENA: + case BPF_MAP_TYPE_INSN_ARRAY: + case BPF_MAP_TYPE_PROG_ARRAY: + case BPF_MAP_TYPE_BLOOM_FILTER: + return true; + default: + return false; + } +} + static int check_map_func_compatibility(struct bpf_verifier_env *env, struct bpf_map *map, int func_id) { if (!map) return 0; + /* + * check_map_prog_compatibility() asks the program. That misses the + * sleepable callback of another program, and an inner map. They may + * use any map under the RCU read lock. Disabled preemption or + * interrupts will not do: the maps ask for rcu_read_lock_held(). + */ + if (in_sleepable(env) && !env->cur_state->active_rcu_locks && + !in_rcu_callback(env) && !map_needs_no_rcu(map)) { + verbose(env, "map '%s' needs bpf_rcu_read_lock() in sleepable context\n", + map->name); + return -EINVAL; + } + /* We need a two way check, first is from map perspective ... */ switch (map->map_type) { case BPF_MAP_TYPE_PROG_ARRAY:
@@ -20676,35 +20722,11 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, return -EINVAL; } - if (prog->sleepable) - switch (map->map_type) { - case BPF_MAP_TYPE_HASH: - case BPF_MAP_TYPE_RHASH: - case BPF_MAP_TYPE_LRU_HASH: - case BPF_MAP_TYPE_ARRAY: - case BPF_MAP_TYPE_PERCPU_HASH: - case BPF_MAP_TYPE_PERCPU_ARRAY: - case BPF_MAP_TYPE_LRU_PERCPU_HASH: - case BPF_MAP_TYPE_LPM_TRIE: - case BPF_MAP_TYPE_ARRAY_OF_MAPS: - case BPF_MAP_TYPE_HASH_OF_MAPS: - case BPF_MAP_TYPE_RINGBUF: - case BPF_MAP_TYPE_USER_RINGBUF: - case BPF_MAP_TYPE_INODE_STORAGE: - case BPF_MAP_TYPE_SK_STORAGE: - case BPF_MAP_TYPE_TASK_STORAGE: - case BPF_MAP_TYPE_CGRP_STORAGE: - case BPF_MAP_TYPE_QUEUE: - case BPF_MAP_TYPE_STACK: - case BPF_MAP_TYPE_ARENA: - case BPF_MAP_TYPE_INSN_ARRAY: - case BPF_MAP_TYPE_PROG_ARRAY: - break; - default: - verbose(env, - "Sleepable programs can only use array, hash, ringbuf and local storage maps\n"); - return -EINVAL; - } + if (prog->sleepable && !map_needs_no_rcu(map)) { + verbose(env, + "Sleepable programs can only use array, hash, ringbuf and local storage maps\n"); + return -EINVAL; + } if (bpf_map_is_cgroup_storage(map) && bpf_cgroup_storage_assign(env->prog->aux, map)) {
--
2.55.0