On Sat Jun 6, 2026 at 5:39 AM PDT, Jiri Olsa wrote:

Adding benchmark test that attaches to (almost) all allowed tracing
functions and display attach/detach times.

  # ./test_progs -t tracing_multi_bench_attach -v
  bpf_testmod.ko is already unloaded.
  Loading bpf_testmod.ko...
  Successfully loaded bpf_testmod.ko.
  serial_test_tracing_multi_bench_attach:PASS:btf__load_vmlinux_btf 0 nsec
  serial_test_tracing_multi_bench_attach:PASS:tracing_multi_bench__open_and_load 0 nsec
  serial_test_tracing_multi_bench_attach:PASS:get_syms 0 nsec
  serial_test_tracing_multi_bench_attach:PASS:bpf_program__attach_tracing_multi 0 nsec
  serial_test_tracing_multi_bench_attach: found 51186 functions
  serial_test_tracing_multi_bench_attach: attached in   1.295s
  serial_test_tracing_multi_bench_attach: detached in   0.243s

...

+	if (!ASSERT_OK(bpf_get_ksyms(&ksyms, true), "get_syms"))
+		goto cleanup;
+
+	/* Get all ftrace 'safe' symbols.. */
+	for (i = 0; i < ksyms->filtered_cnt; i++) {
+		if (!tsearch(&ksyms->filtered_syms[i], &root, compare)) {
+			ASSERT_FAIL("tsearch failed");
+			goto cleanup;
+		}
+	}
+
+	/* ..and filter them through BTF and btf_type_is_traceable_func. */
+	nr = btf__type_cnt(btf);
+	for (type_id = 1; type_id < nr; type_id++) {
+		const struct btf_type *type;
+		const char *str;
+
+		type = btf__type_by_id(btf, type_id);
+		if (!type)
+			break;
+
+		if (BTF_INFO_KIND(type->info) != BTF_KIND_FUNC)
+			continue;
+
+		str = btf__name_by_offset(btf, type->name_off);
+		if (!str)
+			break;
+
+		if (!tfind(&str, &root, compare))
+			continue;
+
+		if (!btf_type_is_traceable_func(btf, type))
+			continue;
+
+		err = libbpf_ensure_mem((void **) &ids, &cap, sizeof(*ids), cnt + 1);
+		if (err)
+			goto cleanup;
+
+		ids[cnt++] = type_id;
+	}

This filtering wasn't enough.
I've added removal of duplicates here while applying:

+       /*
+        * Collect names that are not unique in kallsyms. The kernel resolves a
+        * tracing-multi BTF id to an address with kallsyms_lookup_name(), which
+        * returns the first symbol of that name. For a duplicate name that may
+        * be a different (non-ftrace-able) instance than the ftrace-able one in
+        * available_filter_functions, so attaching to it by BTF id fails with
+        * -ENOENT (e.g. t_start/t_next/t_stop). ksyms->syms is sorted by name,
+        * so equal names are adjacent.
+        */
+       for (i = 1; i < ksyms->sym_cnt; i++) {
+               if (strcmp(ksyms->syms[i].name, ksyms->syms[i - 1].name))
+                       continue;
+               if (!tsearch(&ksyms->syms[i].name, &dups, compare)) {
+                       ASSERT_FAIL("tsearch failed");
+                       goto cleanup;
+               }
+       }


+               /* Skip names that are not unique in kallsyms, see above. */
+               if (tfind(&str, &dups, compare))
+                       continue;


As claude explains it:
----
1. The kernel attaches tracing_multi by BTF id. To get an address it resolves
the BTF function name via kallsyms_lookup_name(tname) and requires
ftrace_location(addr) — kernel/bpf/verifier.c:19380:
addr = kallsyms_lookup_name(tname);
...
if (!addr || !ftrace_location(addr))
        return -ENOENT;
2. t_start/t_next/t_stop each have 5 instances in this kernel. Only one is
ftrace-able — the copies in kernel/trace/* are built notrace (ftrace's Makefile
strips -pg), so only the unrelated copy is in available_filter_functions:
3. kallsyms_lookup_name() returns the lowest-address instance among equal names
(exact strcmp, lowest seq). That instance has no fentry → ftrace_location()
returns 0 → -ENOENT, which aborts the whole all-or-nothing bench attach.

Why the bench includes them: it intersects BTF FUNC names with
available_filter_functions names. Since some t_start is ftrace-able, the name
passes the filter — but the kernel resolves the wrong (non-ftrace-able)
t_start. The author's kernel apparently had the ftrace-able copy at the lowest
address, so it passed there.

This is a pre-existing limitation, not multi-specific: single fentry attach by
BTF id uses the same kallsyms_lookup_name(tname) path (verifier.c:19120) — you
can't reliably fentry-attach to any duplicate-named function on this kernel
either.
----

Maybe we should adjust bpf_get_ksyms() instead. Not sure.

After the fix in my gcc 15 kernel with KASAN:
serial_test_tracing_multi_bench_attach: found 49045 functions
serial_test_tracing_multi_bench_attach: attached in   4.115s
serial_test_tracing_multi_bench_attach: detached in   2.313s