Re: [RFC] store function address in BTF

On Wed, Oct 06, 2021 at 09:17:39AM -0700, Andrii Nakryiko wrote:

On Wed, Oct 6, 2021 at 1:42 AM Jiri Olsa [off-list ref] wrote:

quoted

hi,
I'm hitting performance issue and soft lock ups with the new version
of the patchset and the reason seems to be kallsyms lookup that we
need to do for each btf id we want to attach

I tried to change kallsyms_lookup_name linear search into rbtree search,
but it has its own pitfalls like duplicate function names and it still
seems not to be fast enough when you want to attach like 30k functions

How not fast enough is it exactly? How long does it take?

30k functions takes 75 seconds for me, it's loop calling bpf_check_attach_target

getting soft lock up messages:

krava33 login: [  168.896671] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [bpftrace:1087]

quoted

so I wonder we could 'fix this' by storing function address in BTF,
which would cut kallsyms lookup completely, because it'd be done in
compile time

my first thought was to add extra BTF section for that, after discussion
with Arnaldo perhaps we could be able to store extra 8 bytes after
BTF_KIND_FUNC record, using one of the 'unused' bits in btf_type to
indicate that? or new BTF_KIND_FUNC2 type?

thoughts?

I'm strongly against this, because (besides the BTF bloat reason) we
need similar mass attachment functionality for kprobe/kretprobe and
that one won't be relying on BTF FUNCs, so I think it's better to
stick to the same mechanism for figuring out the address of the
function.

ok

If RB tree is not feasible, we can do a linear search over unsorted
kallsyms and do binary search over sorted function names (derived from
BTF IDs). That would be O(Nlog(M)), where N is number of ksyms, M is
number of BTF IDs/functions-to-be-attached-to. If we did have an RB
tree for kallsyms (is it hard to support duplicates? why?) it could be
even faster O(Mlog(N)).

I had issues with generic kallsyms rbtree in the post some time ago,
I'll revisit it to check on details.. but having the tree with just
btf id functions might clear that.. I'll check

thanks,
jirka

quoted

thanks,
jirka