Edward Cree [off-list ref] writes:

On 17/10/2019 13:11, Toke Høiland-Jørgensen wrote:

quoted

I think there's a conceptual disconnect here in how we view what an XDP
program is. In my mind, an XDP program is a stand-alone entity tied to a
particular application; not a library function that can just be inserted
into another program.

To me, an XDP (or any other eBPF) program is a function that is already
 being 'inserted into another program', namely, the kernel.  It's a
 function that's being wired up to a hook in the kernel.  Which isn't
 so different to wiring it up to a hook in a function that's wired up to
 a hook in the kernel (which is what my proposal effectively does).

Yes, see: Different mental models leading to different notions of what
is the natural way to do things :)

quoted

Setting aside that for a moment; the reason I don't think this belongs
in userspace is that putting it there would carry a complexity cost that
is higher than having it in the kernel.

Complexity in the kernel is more expensive than in userland.  There are
 several reasons for this, such as:
* The kernel's reliability requirements are stricter — a daemon that
  crashes can be restarted, a kernel that crashes ruins your day.
* Userland has libraries available for many common tasks that can't be
  used in the kernel.
* Anything ABI-visible (which this would be) has to be kept forever even
  if it turns out to be a Bad Idea™, because We Do Not Break
Userspace™.

To me, the first and last of those are actually arguments for putting
this into the kernel (from the consuming application's PoV). Surely we
want something that's reliable and well-supported? ;)

The last of these is the big one, and means that wherever possible the
proper course is to prototype functionality in userspace, and then
once the ABI is solid and known-useful, it can move to the kernel if
there's an advantage to doing so

To me, the prototyping was the tail call-based stuff Facebook and
Cloudflare has been doing, and this is an attempt to synthesise that
into something that we can actually agree to support as part of the XDP
feature set.

quoted

- Add a new dependency to using XDP (now you not only need the kernel
  and libraries, you'll also need the daemon).

You'll need *a* daemon. You won't be tied to a specific
implementation.

But the point is that I *want* this to be a specific implementation; or
rather, a specific interface. I.e., I want this to be an interface that
people can rely on being available, rather than have a proliferation of
slightly different ways to achieve this that are subtly incompatible.

quoted

    - Keeping track of which XDP programs are loaded and attached to
      each interface

There's already an API to query this.

There's an API, but the daemon still has to deal with it.

You would probably want an atomic cmpxchg operation, so that you can
detect if someone else is fiddling with XDP and scream noisy warnings.

Yup, probably going to do some sort of atomic program replace API in any
case :)

quoted

    - Some kind of interface with the verifier; if an app does
      xdpd_rpc_load(prog), how is the verifier result going to get back
      to the caller?

The daemon will get the verifier log back when it tries to update the
program; it might want to do a bit of translation before passing it
on, but an RPC call can definitely return errors to the caller.

I wasn't implying that it was impossible to report errors over RPC.
I was saying that you "a bit of translation" is not as trivial as you
make it out to be...

In the Ideal World of kernel dynamic linking, of course, each app prog
gets submitted to the verifier by the app to create a floating function
in the kernel that's not bound to any XDP hook (app gets its verifier
responses at this point)

I believe this is what Alexei means by "indirect calls". That is
different, though, because it implies that each program lives as a
separate object in the kernel - and so it might actually work. What you
were talking about (until this paragraph) was something that was
entirely in userspace, and all the kernel sees is a blob of the eBPF
equivalent of `cat *.so > my_composite_prog.so`.

-Toke