Re: [PATCH bpf-next] libbpf: support module BTF for BPF_TYPE_ID_TARGET CO-RE relocation

On Mon, 7 Dec 2020, Andrii Nakryiko wrote:

On Mon, Dec 7, 2020 at 7:12 PM Alexei Starovoitov
[off-list ref] wrote:

quoted

On Mon, Dec 07, 2020 at 04:38:16PM +0000, Alan Maguire wrote:

quoted

Sorry about this Andrii, but I'm a bit stuck here.

I'm struggling to get tests working where the obj fd is used to designate
the module BTF. Unless I'm missing something there are a few problems:

- the fd association is removed by libbpf when the BPF program has loaded;
the module fds are closed and the module BTF is discarded.  However even if
that isn't done (and as you mentioned, we could hold onto BTF that is in
use, and I commented out the code that does that to test) - there's
another problem:
- I can't see a way to use the object fd value we set here later in BPF
program context; btf_get_by_fd() returns -EBADF as the fd is associated
with the module BTF in the test's process context, not necessarily in
the context that the BPF program is running.  Would it be possible in this
case to use object id? Or is there another way to handle the fd->module
BTF association that we need to make in BPF program context that I'm
missing?
- A more long-term issue; if we use fds to specify module BTFs and write
the object fd into the program, we can pin the BPF program such that it
outlives fds that refer to its associated BTF.  So unless we pinned the
BTF too, any code that assumed the BTF fd-> module mapping was valid would
start to break once the user-space side went away and the pinned program
persisted.

All of the above are not issues. They are features of FD based approach.
When the program refers to btf via fd the verifier needs to increment btf's refcnt
so it won't go away while the prog is running. For module's BTF it means
that the module can be unloaded, but its BTF may stay around if there is a prog
that needs to access it.
I think the missing piece in the above is that btf_get_by_fd() should be
done at load time instead of program run-time.
Everything FD based needs to behave similar to map_fds where ld_imm64 insn
contains map_fd that gets converted to map_ptr by the verifier at load time.

Right. I was going to extend verifier to do the same for all used BTF
objects as part of ksym support for module BTFs. So totally agree.
Just didn't need it so far.

Does this approach prevent more complex run-time specification of BTF 
object fd though?  For example, I've been working on a simple tracer 
focused on kernel debugging; it uses a BPF map entry for each kernel 
function that is traced. User-space populates the map entry with BTF type 
ids for the function arguments/return value, and when the BPF program 
runs it uses the instruction pointer to look up the map entry for that
function, and uses bpf_snprintf_btf() to write the string representations 
of the function arguments/return values.  I'll send out an RFC soon, 
but longer-term I was hoping to extend it to support module-specific 
types.  Would a dynamic case like that - where the BTF module fd is looked 
up in a map entry during program execution (rather than derived via 
__btf_builtin_type_id()) work too? Thanks!

Alan