Re: [PATCH v2] cgroup/bpf: fast path for not loaded skb BPF filtering
flat view
From: Martin KaFai Lau <hidden>
Date: 2021-12-14 07:27:44
Also in:
bpf, lkml
On Sat, Dec 11, 2021 at 07:17:49PM +0000, Pavel Begunkov wrote:
cgroup_bpf_enabled_key static key guards from overhead in cases where no cgroup bpf program of a specific type is loaded in any cgroup. Turn out that's not always good enough, e.g. when there are many cgroups but ones that we're interesting in are without bpf. It's seen in server environments, but the problem seems to be even wider as apparently systemd loads some BPF affecting my laptop. Profiles for small packet or zerocopy transmissions over fast network show __cgroup_bpf_run_filter_skb() taking 2-3%, 1% of which is from migrate_disable/enable(), and similarly on the receiving side. Also got +4-5% of t-put for local testing.
What is t-put? throughput? Local testing means sending to lo/dummy? [ ... ]
quoted hunk ↗ jump to hunk
diff --git a/include/linux/bpf-cgroup.h b/include/linux/bpf-cgroup.h index 11820a430d6c..793e4f65ccb5 100644 --- a/include/linux/bpf-cgroup.h +++ b/include/linux/bpf-cgroup.h@@ -219,11 +219,28 @@ int bpf_percpu_cgroup_storage_copy(struct bpf_map *map, void *key, void *value); int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key, void *value, u64 flags); +static inline bool +__cgroup_bpf_prog_array_is_empty(struct cgroup_bpf *cgrp_bpf, + enum cgroup_bpf_attach_type type)
Lets remove this.
quoted hunk ↗ jump to hunk
+{ + struct bpf_prog_array *array = rcu_access_pointer(cgrp_bpf->effective[type]); + + return array == &empty_prog_array.hdr; +} + +#define CGROUP_BPF_TYPE_ENABLED(sk, atype) \
and change cgroup.c to directly use this instead, so everywhere holding a fullsock sk will use this instead of having two helpers for empty check. [ ... ]
quoted hunk ↗ jump to hunk
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2405e39d800f..fedc7b44a1a9 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c@@ -1967,18 +1967,10 @@ static struct bpf_prog_dummy { }, }; -/* to avoid allocating empty bpf_prog_array for cgroups that - * don't have bpf program attached use one global 'empty_prog_array' - * It will not be modified the caller of bpf_prog_array_alloc() - * (since caller requested prog_cnt == 0) - * that pointer should be 'freed' by bpf_prog_array_free() - */ -static struct { - struct bpf_prog_array hdr; - struct bpf_prog *null_prog; -} empty_prog_array = { +struct bpf_empty_prog_array empty_prog_array = { .null_prog = NULL, }; +EXPORT_SYMBOL(empty_prog_array);
nit. Since it is exported, may be prefix it with 'bpf_'.