[PATCH bpf-next 22/38] bpf: Count sleepable callbacks as sleepable map users
flat view
COOLING6d
SUPERSEDED
From: Alexei Starovoitov <hidden>
Date: 2026-10-17 01:29:36
Also in:
bpf
Subsystem:
bpf [core], bpf [general] (safe dynamic programs and tools), the rest · Maintainers:
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman, Kumar Kartikeya Dwivedi, Linus Torvalds
Revision v1 of 2 in this series.
Revisions (2)
- v1 current
- v2 [diff vs current]
From: Josef Bacik <josef@toxicpanda.com>
map->sleepable_refcnt tells bpf_map_fd_put_ptr() whether an inner map that
is taken out of its outer map has to wait for an RCU Tasks Trace grace
period as well. It counts the sleepable programs that use the outer map.
The bpf_wq or bpf_task_work callback of a program that is not sleepable
runs under rcu_read_lock_trace() only, and is not counted.
Such a callback can look an inner map up and keep it while user space
deletes it from the outer map. The inner map is freed behind a plain RCU
grace period, under the callback:
BUG: KASAN: slab-use-after-free in array_map_update_elem+0x6bc/0x7d0
Count a program with such a callback. The maps are bound before the
verifier meets the callback, so raise the count of those that are bound
when it does.
Fixes: 8e83da9732d9 ("bpf: add bpf_wq_start")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/core.c | 2 +-
kernel/bpf/syscall.c | 2 +-
kernel/bpf/verifier.c | 17 +++++++++++++++--
3 files changed, 17 insertions(+), 4 deletions(-)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 2154c18a5a4d..e98c79e8009c 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c@@ -3044,7 +3044,7 @@ void __bpf_free_used_maps(struct bpf_prog_aux *aux, bool sleepable; u32 i; - sleepable = aux->prog->sleepable; + sleepable = bpf_prog_runs_sleepable(aux->prog); for (i = 0; i < len; i++) { map = used_maps[i]; if (map->ops->map_poke_untrack)
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 755529888a72..3f215f2509b2 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c@@ -6253,7 +6253,7 @@ static int bpf_prog_bind_map(union bpf_attr *attr) /* The bpf program will not access the bpf map, but for the sake of * simplicity, increase sleepable_refcnt for sleepable program as well. */ - if (prog->sleepable) + if (bpf_prog_runs_sleepable(prog)) atomic64_inc(&map->sleepable_refcnt); memcpy(used_maps_new, used_maps_old, sizeof(used_maps_old[0]) * prog->aux->used_map_cnt);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 6b15862a15e8..1f2a846ae415 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c@@ -593,6 +593,19 @@ static bool is_async_cb_sleepable(struct bpf_verifier_env *env, struct bpf_insn return false; } +/* From here on the maps of the program count it as a sleepable user. */ +static void mark_sleepable_async_cb(struct bpf_verifier_env *env) +{ + int i; + + if (bpf_prog_runs_sleepable(env->prog)) + return; + + env->prog->aux->sleepable_async_cb = true; + for (i = 0; i < env->used_map_cnt; i++) + atomic64_inc(&env->used_maps[i]->sleepable_refcnt); +} + bool bpf_is_may_goto_insn(struct bpf_insn *insn) { return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_MAY_GOTO;
@@ -11212,7 +11225,7 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins if (IS_ERR(async_cb)) return PTR_ERR(async_cb); if (is_async_cb_sleepable(env, insn)) - env->prog->aux->sleepable_async_cb = true; + mark_sleepable_async_cb(env); callee = async_cb->frame[0]; callee->async_entry_cnt = state->frame[0]->async_entry_cnt + 1;
@@ -20776,7 +20789,7 @@ static int __add_used_map(struct bpf_verifier_env *env, struct bpf_map *map) if (err) return err; - if (env->prog->sleepable) + if (bpf_prog_runs_sleepable(env->prog)) atomic64_inc(&map->sleepable_refcnt); /* hold the map. If the program is rejected by verifier,
--
2.55.0