Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

On Fri, Aug 7, 2020 at 10:24 AM Jean-Philippe Brucker
[off-list ref] wrote:

Hi,

[Adding the linux-arm-kernel list on Cc]

On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote:

quoted

Hi everyone,

recently we have begun extensive research into eBPF and related
technologies. Seeking an easier development process, we have switched over
to using the eBPF CO-RE [0] approach internally which has enabled us to
simplify most aspects of eBPF development, especially those related to
cross-compilation.

However, as part of these efforts we have stumbled upon several problems
that we feel would benefit from a community discussion where we may share
our solutions and discuss alternatives moving forward.

As a reference point, we have started researching and modifying several eBPF
CO-RE samples that have been developed or migrated from existing `bcc`
tooling. Most notable examples are those present in `bcc`'s `libbpf-tools`
directory [1]. Some of these samples have just recently been converted to
respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has
proven to be very interesting.

First showstopper for cross-compiling aforementioned example on the ARM
32-bit platform has been with regards to generation of the required
`vmlinux.h` kernel header from the BTF information. More specifically, our
initial approach to have e.g. a compilation target dependency which would
invoke `bpftool` at configure time was not appropriate due to several
issues: a) CO-RE requires host kernel to have been compiled in such a way to
expose BTF information which may not available, and b) the generated
`vmlinux.h` was actually architecture-specific.

The second point proved interesting because `tcpconnect` makes use of the
`BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs *ctx` as
the first function parameter. The `pt_regs` structure is defined by the
kernel and is architecture-specific. Since `libbpf` does have
architecture-specific conditionals, pairing it with an "invalid" `vmlinux.h`
resulted in cross-compilation failure as `libbpf` provided macros that work
with ARM `pt_regs`, and `vmlinux.h` had an x86 `pt_regs` definition. To
resolve this issue, we have resorted to including pre-generated
`<arch>_vmlinux.h` files in our CO-RE build system.

However, there are certainly drawbacks to this approach: a) (relatively)
large file size of the generated headers, b) regular maintenance to
re-generate the header files for various architectures and kernel versions,
and c) incompatible definitions being generated, to name a few. This last
point relates to the the fact that our `aarch64`/`arm64` kernel generates
the following definition using `bpftool`, which has resulted in compilation
failure:

typedef __Poly8_t poly8x16_t[16];

AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
have opted to comment out this line as there was no additional `poly8x16_t`
usage in the header file.

It looks like this "__Poly8_t" type is internal to GCC (provided in
arm_neon.h) and clang has its own internals. I managed to reproduce this
with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
the moment. Maybe libbpf should generate defines to translate these
intrinsics between clang and gcc? Not very elegant. I'll take another
look next week.

libbpf is already blacklisting __builtin_va_list for GCC, so we can
just add __Poly8_t to the list. See [0].
Are there any other types like that? If you guys can provide me this,
I'll gladly update libbpf to take those compiler-provided
types/built-ins into account.

  [0] https://github.com/torvalds/linux/blob/master/tools/lib/bpf/btf_dump.c#L585-L598

quoted

Given various issues we have encountered so far (among which is a kernel
panic/crash on a specific device), additional input and feedback regarding
cross-compilation of the eBPF utilities would be greatly appreciated.

I don't know if there is a room for improvement regarding your a) and b)
points, as I think the added complexity is inherent to cross-building. But
kernel crashes definitely need to be fixed, as well as the above problem.

Thanks,
Jean

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel