Re: How to orchestrate multiple XDP programs

"Brian G. Merrell" [off-list ref] writes:

quoted

Yeah, so what I was thinking was whether it would be useful to, for
instance, define a "bundle" format that contains the config data that
libxdp will understand. Could just be a JSON schema containing keys for
priority, chain call actions and a filename, so you can just point
xdp_program__open_file() at that and it will do the rest.

However, I'm still not quite sure I'm convinced that this will be
generally useful. As you say, you can just as well just set the values
programmatically after loading the file, and I suspect that different
deployments will end up having too much custom stuff around this that
they'll bother using such a facility anyway. WDYT?

Yeah, my hunch is that different deployments will have custom config
data and then getting the data into libxdp format will just be another
data conversion step. I could be wrong.

If it is decided to create a data format, then the JSON schema you
described is pretty much exactly what I had in mind, too. For whatever
that is worth!

Right, I'll mull this over some more but let's defer it for now.

quoted

quoted

I know in a previous e-mail you mentioned having a config file with
priority overrides. That's just not a use case that our team would want
to use. And, my opinion would be that the program using libxdp should be
the one to implement that sort of policy; it keeps libxdp more simple
without needing to worry about parsing config files (and handling config
version changes in the code and the spec). For example, xdp-loader could
have a config file with priority overrides and people could use that
code if they wanted to do something similar.

Yeah, that's totally what would make sense for your deployment case. The
design where libxdp reads a config file comes from my distro
perspective: We want to build a system whereby different applications
can each incorporate XDP functionality and co-exist; and the goal is to
make libxdp the synchronisation point between them. I.e., we can say to
application authors "just use libxdp when writing your application and
it'll work", while at the same time empowering sysadmins to change the
default application ordering.

Ah, yes, that perspective helps a lot; I understand much better why you
would want a centralized configuration file. My mind is actually kind of
melting now thinking about all the use cases.

Is the idea that the configuration file would have the final word? For
example, if there are multiple applications using libxdp (and possibly
even programatically overriding their priorities), could a sysadmin then
write a config file to dictate what they actually want? That would made
sense to me. If that's the goal, then I guess I would take back my
opinion about that policy not belonging in libxdp. I think application
writers would be less likely to use the configuration format (for
reasons we discussed above), but it does seem like a necessary mechanism
for sysadmins to orchestrate multiple applications that don't know about
each other.

Yeah, that would be the idea.

For my team's use case, we have the luxury of being the One
Application to rule all BPF programs.

Yup, obviously being in full control of the system gives you some
options we don't quite have as a distro. But see below - I still think
there are some similarities.

quoted

By having that configuration be part of the library, applications can be
free to use either the command-line loader or include the loading into
their own user-space binary.

But since you are (notionally) both the application developer and system
owner, that is less of a concern for you as you control the whole stack.

quoted

Hopefully I'm even making sense, but like I said, I don't have strong
feelings about the format, as long as we are able to achieve our
required use case of programmatically setting the run configuration
values from a libxdp user program.

Sure, that you can certainly achieve with implementing what libxdp
includes today. I'm just trying to make sure we explore any
opportunities for standardising something useful so others can benefit
from it as well; so I hope you'll forgive my probing :)

8< snip

quoted

quoted

We explicitly do not want defaults set by program authors. We want that
policy to be completely in the hands of the orchestration environment.

Right, OK. How does the admin configuring the orchestration system
figure out which order to run programs in, BTW? Is this obvious from the
nature of the programs, or do you document it out of band somewhere, or
something like that?

We're a pretty huge organization... lots of DCs, public cloud, private
cloud, different kernel versions, sister companies, hundreds of
applications, etc. We want anyone to be able to write cool BPF
programs and userspace applications without needing awareness of
what's running before or after or if that order might change in the
future. I'm sure the desired order will be more obvious for some
programs than others, but we have administrators that can analyze the
BPF programs, compose multiple BPF programs together, and order and
reorder them. We have a team of people that can work with teams to
resolve any interdependencies if necessary.

As an example, we've done something similar for HTTP ingress and
egress Lua plugins in the past. We have dozens of teams that write Lua
code to do custom L7 things with HTTP requests and responses, and then
we have a UI where admins/ops folks can literally drag and drog the
plugins into the desired order. We wouldn't want teams making
assumptions about what order plugins should run in, either.

See, so this is the part that's actually analogous to what we want to do
as a distro. Except the people writing the cool BPF programs are
different software vendors and open source projects, not different
divisions within the same sprawling org. But in a sense the situation is
quite similar.

So thinking a bit more about the difference between your orchestration
system and the model I've been working from, I think the biggest
difference is not that you are assuming control of a system with an
orchestration system. In a sense a distro is also an orchestration
system bringing together different software from different sources.

No, I think the main difference is that in the model you described,
you're assuming that your orchestration system would install the XDP
program on behalf of the application as well as launch the userspace
bits. Whereas I'm assuming that an application that uses XDP will start
in userspace (launched by systemd, most likely), and will then load its
own XDP program after possibly doing some initialisation first (e.g.,
pre-populating maps, that sort of thing).

From what I've understood from what you explained about your setup, your
model could work with both models as well; so why are you assuming that
applications won't want to install their own XDP programs? :)

-Toke