Re: [RFC PATCH bpf-next 0/3] bpf: add tracing for XDP programs using the BPF_PROG_TEST_RUN API

On Thu, Apr 30, 2020 at 7:44 PM Masami Hiramatsu [off-list ref] wrote:

On Tue, 28 Apr 2020 09:19:47 -0300
Arnaldo Carvalho de Melo [off-list ref] wrote:

quoted

Em Tue, Apr 28, 2020 at 12:47:53PM +0200, Eelco Chaudron escreveu:

quoted

On 28 Apr 2020, at 6:04, Alexei Starovoitov wrote:

quoted

On Fri, Apr 24, 2020 at 02:29:56PM +0200, Eelco Chaudron wrote:

quoted

quoted

quoted

quoted

But in reality I think few kprobes in the prog will be enough to
debug the program and XDP prog may still process millions of
packets because your kprobe could be in error path and the user
may want to capture only specific things when it triggers.

quoted

quoted

quoted

quoted

kprobe bpf prog will execute in such case and it can capture
necessary state from xdp prog, from packet or from maps that xdp
prog is using.

quoted

quoted

quoted

quoted

Some sort of bpf-gdb would be needed in user space.  Obviously
people shouldn't be writing such kprob-bpf progs that debug
other bpf progs by hand. bpf-gdb should be able to generate them
automatically.

quoted

quoted

quoted

See my opening comment. What you're describing here is more when
the right developer has access to the specific system. But this
might not even be possible in some environments.

quoted

quoted

All I'm saying that kprobe is a way to trace kernel.
The same facility should be used to trace bpf progs.

quoted

perf doesn’t support tracing bpf programs, do you know of any tools that
can, or you have any examples that would do this?

I'm discussing with Yonghong and Masami what would be needed for 'perf
probe' to be able to add kprobes to BPF jitted areas in addition to
vmlinux and modules.

At a grance, at first we need a debuginfo which maps the source code and
BPF binaries. We also need to get a map from the kernel indicating
which instructions the bpf code was jited to.
Are there any such information?

it's already there. Try 'bpftool prog dump jited id N'
It will show something like this:
; data = ({typeof(errors.leaf) *leaf =
bpf_map_lookup_elem_(bpf_pseudo_fd(1, -11), &type_key); if (!leaf) {
bpf_map_update_elem_(bpf_pseudo_fd(1, -11), &type_key, &zero,
BPF_NOEXIST); leaf = bpf_map_lookup_elem_(bpf_pseudo_fd(1, -11), &t;
 81d:    movabs $0xffff8881a0679000,%rdi
; return bpf_map_lookup_elem((void *)map, key);
 827:    mov    %rbx,%rsi
 82a:    callq  0xffffffffe0f7f448
 82f:    test   %rax,%rax
 832:    je     0x0000000000000838
 834:    add    $0x40,%rax
; if (!data)
 838:    test   %rax,%rax
 83b:    je     0x0000000000000846
 83d:    mov    $0x1,%edi
; lock_xadd(data, 1);
 842:    lock add %edi,0x0(%rax)

Also, I would like to know the target BPF (XDP) is running in kprobes
context or not. BPF tracer sometimes use the kprobes to hook the event
and run in the kprobe (INT3) context. That will be need more work to
probe it.
For the BPF code which just runs in tracepoint context, it will be easy
to probe it. (we may need to break a limitation of notrace, which we
already has a kconfig)

yeah. this mechanism won't be able to debug bpf progs that are
attached to kprobes via int3. But that is rare case.
Most kprobe+bpf are for function entry and adding int3 to jited bpf code
will work just like for normal kernel functions.