[PATCH bpf-next v2 23/43] bpf: Count sleepable callbacks as sleepable map users
flat view
WARM2d
IN BPF-NEXT
From: Alexei Starovoitov <hidden>
Date: 2026-10-17 16:40:08
Also in:
bpf
Subsystem:
bpf [core], bpf [general] (safe dynamic programs and tools), the rest · Maintainers:
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman, Kumar Kartikeya Dwivedi, Linus Torvalds
Revision v2 of 2 in this series; queued in bpf-next as 927c9c985f3e on 2026-10-17.
Revisions (2)
- v1 [diff vs current]
- v2 current
From: Josef Bacik <josef@toxicpanda.com>
map->sleepable_refcnt tells bpf_map_fd_put_ptr() whether an inner map that
is taken out of its outer map has to wait for an RCU Tasks Trace grace
period as well. It counts the sleepable programs that use the outer map.
The bpf_wq or bpf_task_work callback of a program that is not sleepable
runs under rcu_read_lock_trace() only, and is not counted.
Such a callback can look an inner map up and keep it while user space
deletes it from the outer map. The inner map is freed behind a plain RCU
grace period, under the callback:
BUG: KASAN: slab-use-after-free in array_map_update_elem+0x6bc/0x7d0
Count a program with such a callback. The maps are bound before the
verifier meets the callback, so raise the count of those that are bound
when it does.
Fixes: 8e83da9732d9 ("bpf: add bpf_wq_start")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/core.c | 2 +-
kernel/bpf/syscall.c | 2 +-
kernel/bpf/verifier.c | 17 +++++++++++++++--
3 files changed, 17 insertions(+), 4 deletions(-)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 69285abcf594..e1216927373a 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c@@ -3042,7 +3042,7 @@ void __bpf_free_used_maps(struct bpf_prog_aux *aux, bool sleepable; u32 i; - sleepable = aux->prog->sleepable; + sleepable = bpf_prog_runs_sleepable(aux->prog); for (i = 0; i < len; i++) { map = used_maps[i]; if (map->ops->map_poke_untrack)
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 30d570c1c9a0..c412bcd1903f 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c@@ -6257,7 +6257,7 @@ static int bpf_prog_bind_map(union bpf_attr *attr) /* The bpf program will not access the bpf map, but for the sake of * simplicity, increase sleepable_refcnt for sleepable program as well. */ - if (prog->sleepable) + if (bpf_prog_runs_sleepable(prog)) atomic64_inc(&map->sleepable_refcnt); memcpy(used_maps_new, used_maps_old, sizeof(used_maps_old[0]) * prog->aux->used_map_cnt);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 6b15862a15e8..1f2a846ae415 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c@@ -593,6 +593,19 @@ static bool is_async_cb_sleepable(struct bpf_verifier_env *env, struct bpf_insn return false; } +/* From here on the maps of the program count it as a sleepable user. */ +static void mark_sleepable_async_cb(struct bpf_verifier_env *env) +{ + int i; + + if (bpf_prog_runs_sleepable(env->prog)) + return; + + env->prog->aux->sleepable_async_cb = true; + for (i = 0; i < env->used_map_cnt; i++) + atomic64_inc(&env->used_maps[i]->sleepable_refcnt); +} + bool bpf_is_may_goto_insn(struct bpf_insn *insn) { return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_MAY_GOTO;
@@ -11212,7 +11225,7 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins if (IS_ERR(async_cb)) return PTR_ERR(async_cb); if (is_async_cb_sleepable(env, insn)) - env->prog->aux->sleepable_async_cb = true; + mark_sleepable_async_cb(env); callee = async_cb->frame[0]; callee->async_entry_cnt = state->frame[0]->async_entry_cnt + 1;
@@ -20776,7 +20789,7 @@ static int __add_used_map(struct bpf_verifier_env *env, struct bpf_map *map) if (err) return err; - if (env->prog->sleepable) + if (bpf_prog_runs_sleepable(env->prog)) atomic64_inc(&map->sleepable_refcnt); /* hold the map. If the program is rejected by verifier,
--
2.55.0