On Mon, Jun 12, 2023 at 10:25:55PM +0200, Jiri Olsa wrote:

On Mon, Jun 12, 2023 at 11:02:22AM -0400, Steven Rostedt wrote:

quoted

On Mon, 12 Jun 2023 07:49:53 -0700
Yonghong Song [off-list ref] wrote:

quoted

I am actually interested in how available_filter_functions_addrs
will be used. For example, bpf_program__attach_kprobe_multi_opts()
can already take addresses from kallsyms. How to use
available_filter_functions_addrs to facilitate kprobe_multi?

the problem is that we need to do 2 passes:

 - through available_filter_functions and find out if the function is traceable
 - through /proc/kallsyms to get the address for traceable function

having available_filter_functions symbols together with addresses allow
us to skip the kallsyms step

and we are ok with the address in available_filter_functions_addr not being the
function entry, because kprobe_multi uses fprobe and that handles both entry and
patch-site address properly

quoted

quoted

Do we need to change kernel APIs? It would be great at least we
got a RFC patch to answer these questions.

I agree, having that information would also be useful to me.

Jiri? Andrii?

so we have 2 interfaces how to create kprobe_multi link:

  a) passing symbols to kernel

     1) user gathers symbols and need to ensure that they are
        trace-able -> pass through available_filter_functions file

     2) kernel takes those symbols and translates them to addresses
        through kallsyms api

     3) addresses are passed to fprobe/ftrace through:

         register_fprobe_ips
         -> ftrace_set_filter_ips

  b) passing addresses to kernel

     1) user gathers symbols and needs to ensure that they are
        trace-able -> pass through available_filter_functions file

     2) user takes those symbols and translates them to addresses
       through /proc/kallsyms

     3) addresses are passed to the kernel and kernel calls:

         register_fprobe_ips
         -> ftrace_set_filter_ips


The new available_filter_functions_addrs file helps us with option b),
because we can make 'b 1' and 'b 2' in one step - while filtering traceable
functions, we get the address directly.

I tested the new available_filter_functions_addrs changes with some hacked
selftest changes, you can check it in here [1].

I assume Jackie Liu will send new version of her patchset [2] based on this
new available_filter_functions_addrs file.

I think we should have these changes coming together and add some perf
measurements from before and after to make the benefit apparent.

FYI I did some perf meassurements and the speedup is not substantial :-\

looks like the symbols resolving to addresses we do in kernel for kprobe_multi
link is more faster/cheaper than I thought 

but still there is 'some' speedup and we will get rid of the extra
/proc/kallsyms parsing, so I think it's still worth it to have the
new file


base:

 Performance counter stats for './test_progs -n 103/1':

       103,423,103      cycles:u
    79,279,231,029      cycles:k
    79,382,694,663      cycles

      18.627593589 seconds time elapsed

       0.025999000 seconds user
      18.323855000 seconds sys


with fix:

 Performance counter stats for './test_progs -n 103/1':

       126,659,572      cycles:u
    77,951,768,179      cycles:k
    78,078,467,451      cycles

      18.651464273 seconds time elapsed

       0.025001000 seconds user
      18.243828000 seconds sys


jirka