On Fri, Nov 6, 2020 at 1:00 AM Jiri Benc [off-list ref] wrote:

On Thu, 5 Nov 2020 12:45:39 -0800, Andrii Nakryiko wrote:

quoted

That's not true. If you need new functionality like BTF, CO-RE,
function-by-function verification, etc., then yes, you have to update
kernel, compiler, libbpf, sometimes pahole. But if you have an BPF
application that doesn't use and need any of the newer features, it
will keep working just fine with the old kernel, old libbpf, and old
compiler.

I'm fine with this.

It doesn't work that well in practice, we've found ourselves chasing
problems caused by llvm update (problems for older bpf programs, not
new ones), problems on non-x86_64 caused by kernel updates, etc. It can
be attributed to living on the edge and it should stabilize over time,
hopefully. But it's still what the users are experiencing and it's
probably what David is referring to. I expect it to smooth itself over
time.

It's definitely going to be better over time, of course. I honestly
can't remember many cases where working applications stopped working
with newer kernels. I only remember cases when Clang changed the code
generation patterns. Also there were few too permissive checks fixed
in later kernels, which could break apps, if apps relied on buggy
logic. That did happen I think.

But anyway, I bet people just got a "something like that happened in
the past" flag in their head, but won't be able to recall specific
details anymore. My point is that we (BPF developers) don't take these
things lightly, so I'd just like to avoid the perception that we don't
care about this. Because we do, despite it sometimes being painful.
But there are layers upon layers of abstraction and it's not all
always under our control, so things might break.

Add to that the fact that something that is in fact a new feature is
perceived as a bug fix by some users. For example, a perfectly valid
and simple C program, not using anything shiny but a basic simple loop,
compiles just fine but is rejected by the kernel. A newer kernel and a
newer compiler and a newer libbpf and a newer pahole will cause the
same program to be accepted. Now, the user does not see that for this,
a new load of BTF functionality had to be added and all those mentioned
projects enhanced with substantial code. All they see is their simple
hello world test program did not work and now it does.

Right. The unavoidable truth that anyone using BPF has to have at
least a surface-level idea about what BPF verifier is and what (and
sometimes how) it checks. It also gets better over time so much that
for some simpler application it will just work perfectly from the
first version of written code.

But let's also not lose perspective here. There aren't many examples
of practical static verification of program safety and termination,
right? It's tricky, and especially when making it also practical for a
wide variety of use cases.

I'm not saying I have a solution nor I'm saying you should do something
about it. Just trying to explain the perception.

Thanks for that, it's a good perspective. Hopefully my explanation
also makes sense ;)

 Jiri