On 21/02/17 04:53PM, Toke Høiland-Jørgensen wrote:

"Brian G. Merrell" [off-list ref] writes:

quoted

On 21/02/15 01:47PM, Toke Høiland-Jørgensen wrote:

quoted

"Brian G. Merrell" [off-list ref] writes:

quoted

On 21/02/11 12:18PM, Toke Høiland-Jørgensen wrote:

quoted

"Brian G. Merrell" [off-list ref] writes:

8< snip

quoted

OK, what currently happens is we have a separate, centralized Go web
service exposing an HTTP based API. When the sysadmin calls that API it
stores the config data in a database. Then, we have another service that
periodically queries the database and writes the config data to a
constant database (cdb) and stores that in blob storage. Then, there is
a service running on each node that periodically pulls down the latest
cdb. Our orchestration tool running on each node is watching for new
cdbs using inotify; when the tool sees a new CDB it loads the new
configuration data--which, for us, literally ends up just being JSON
data--and does anything that needs done.

I had omitted those details for a couple of reasons: First, it's kind of a
lot and I didn't know it would be helpful. Second, this is the way it
works currently because, for expediency, we leveraged the internal
ecosystem that was already setup. We will likely move away from it, at
least partially.

So, I think the important part is that our orchestration tool will
periodically get the config data in JSON format. A path to each BPF
program is in the config data and the orchestration tool downloads them
as needed. We may move to just including the BPF program binary in the
config data--TBD. Obviously, we aren't using libxdp yet, so our config
data doesn't have "run priority", instead the config data has the order
the BPF programs need to run, and BPF programs themselves have to do the
bpf_tail_call (and the orchestration tool does a bunch of complicated
orchestration to get the chain in the right order). The config data also
contains a bunch of other information to do the orchestration, e.g.,
interface, ingress or egress, tc or xdp, what userspace code to run and
any config values for that, etc.

Hopefully that answers your question, and sorry if it was too much
information :)

No, that was very helpful, thanks! Just the kind of detail I was after
to understand your deployment scenario :)

OK, phew :)

quoted

quoted

quoted

I was planning to set the run order programatically on the XDP program
objects via libxdp calls. It looks like your libxdp implementation
already has ways to do this in the form of xdp_program__set_run_prio()
and xdp_program__chain_call_enabled().

Does that make sense? This is still all very theoretical for me at this
point!

Yup, totally possible to set this programmatically with libxdp as well
today. However, before doing so you still need to communicate the list
of BPF programs and their run configuration to each node. And I'm
thinking it may be worthwhile to specify how to do this as part of the
"protocol" and also teach libxdp about the format, so others won't have
to reinvent the same thing later.

It seems like I must be missing something here, but my plan was to do
this all programmatically by calling libxdp functions from the
orchestration tool by 1) calling something like xdp_program__open_file()
to load the XDP program, and then 2) setting the run configuration by
calling something like xdp_program__set_run_prio() and
xdp_program__chain_call_enabled(), and 3) adding the programs to the
dispatcher and loading it.

Correct me if I'm wrong, but I guess you're saying that it might be
worth creating an abstraction in libxdp where a user can pass in the
necessary config data and libxdp does the work, that I just summarized
in the previous paragraph, on the user's behalf. I can see how that
could be a useful abstraction.

Yeah, so what I was thinking was whether it would be useful to, for
instance, define a "bundle" format that contains the config data that
libxdp will understand. Could just be a JSON schema containing keys for
priority, chain call actions and a filename, so you can just point
xdp_program__open_file() at that and it will do the rest.

However, I'm still not quite sure I'm convinced that this will be
generally useful. As you say, you can just as well just set the values
programmatically after loading the file, and I suspect that different
deployments will end up having too much custom stuff around this that
they'll bother using such a facility anyway. WDYT?

Yeah, my hunch is that different deployments will have custom config
data and then getting the data into libxdp format will just be another
data conversion step. I could be wrong.

If it is decided to create a data format, then the JSON schema you
described is pretty much exactly what I had in mind, too. For whatever
that is worth!

quoted

I know in a previous e-mail you mentioned having a config file with
priority overrides. That's just not a use case that our team would want
to use. And, my opinion would be that the program using libxdp should be
the one to implement that sort of policy; it keeps libxdp more simple
without needing to worry about parsing config files (and handling config
version changes in the code and the spec). For example, xdp-loader could
have a config file with priority overrides and people could use that
code if they wanted to do something similar.

Yeah, that's totally what would make sense for your deployment case. The
design where libxdp reads a config file comes from my distro
perspective: We want to build a system whereby different applications
can each incorporate XDP functionality and co-exist; and the goal is to
make libxdp the synchronisation point between them. I.e., we can say to
application authors "just use libxdp when writing your application and
it'll work", while at the same time empowering sysadmins to change the
default application ordering.

Ah, yes, that perspective helps a lot; I understand much better why you
would want a centralized configuration file. My mind is actually kind of
melting now thinking about all the use cases.

Is the idea that the configuration file would have the final word? For
example, if there are multiple applications using libxdp (and possibly
even programatically overriding their priorities), could a sysadmin then
write a config file to dictate what they actually want? That would made
sense to me. If that's the goal, then I guess I would take back my
opinion about that policy not belonging in libxdp. I think application
writers would be less likely to use the configuration format (for
reasons we discussed above), but it does seem like a necessary mechanism
for sysadmins to orchestrate multiple applications that don't know about
each other. For my team's use case, we have the luxury of being the One
Application to rule all BPF programs.

By having that configuration be part of the library, applications can be
free to use either the command-line loader or include the loading into
their own user-space binary.

But since you are (notionally) both the application developer and system
owner, that is less of a concern for you as you control the whole stack.

quoted

Hopefully I'm even making sense, but like I said, I don't have strong
feelings about the format, as long as we are able to achieve our
required use case of programmatically setting the run configuration
values from a libxdp user program.

Sure, that you can certainly achieve with implementing what libxdp
includes today. I'm just trying to make sure we explore any
opportunities for standardising something useful so others can benefit
from it as well; so I hope you'll forgive my probing :)

8< snip

quoted

We explicitly do not want defaults set by program authors. We want that
policy to be completely in the hands of the orchestration environment.

Right, OK. How does the admin configuring the orchestration system
figure out which order to run programs in, BTW? Is this obvious from the
nature of the programs, or do you document it out of band somewhere, or
something like that?

We're a pretty huge organization... lots of DCs, public cloud, private
cloud, different kernel versions, sister companies, hundreds of
applications, etc. We want anyone to be able to write cool BPF programs
and userspace applications without needing awareness of what's running
before or after or if that order might change in the future. I'm sure
the desired order will be more obvious for some programs than others,
but we have administrators that can analyze the BPF programs, compose
multiple BPF programs together, and order and reorder them. We have a
team of people that can work with teams to resolve any interdependencies
if necessary.

As an example, we've done something similar for HTTP ingress and egress
Lua plugins in the past. We have dozens of teams that write Lua code to
do custom L7 things with HTTP requests and responses, and then we have a
UI where admins/ops folks can literally drag and drog the plugins into
the desired order. We wouldn't want teams making assumptions about what
order plugins should run in, either.

Hopefully that helps!

Thanks again,
Brian