My only concern is that is there any way to reuse flower
code instead
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
of duplicating most of them? Especially when you
specifically mentioned
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
flower2 has the same user-space syntax as flower, this
makes code
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
reusing more reasonable.
Exactly. I believe it is wrong to introduce new classifier
which would
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
basically behave exacly the same as flower, only has
different parser
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
implementation under the hood.
Could you please explore the possibility to replace
flow_dissector by
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
your dissector optionally at first (kernel config for
example)? And I'm
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
not talking only about flower, but about the rest of the
flow_dissector
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
users too.
+1
Does the existing BPF flow dissector not work for some reason? If its
purely
quoted
quoted
a hardware mapping problem, couple questions below.
Hi John,
eBPF in its current form is un-acceleratable is a primary problem,
We have no disagreement here. Offloading a general purpose instruction
set into an architecture (switch, etc.) that looks nothing like this
is a losing game.
quoted
however an eBPF flow dissector would still have the same issues in
complexity and manageability that the kernel flow dissector. PANDA
PANDA is a DSL and runtime, similarly P4 is another DSL and runtime
environment. P4 can compile to BPF, PANDA can as well. I don't think
PANDA or P4 belong in kernel. We can argue about preferred DSLs but
I don't think that is a @netdev @bpf concern.
quoted
pretty much can address that since the same source code for flow
dissector used in tc-flower could be compiled into eBPF and used with
TC (that gives extensibility at some incremental performance
degradation). There is an interesting caveat there in that to satisfy
verifier we had to break up the parser to avoid being flagged for
complexity. The parser code however already started with a restricted
structure that enforces rules that prevent the issues the verified is
checking for, when we get to verifier though it doesn't have any that
context and hence views it as general purpose code. Introducing domain
specific constructs into eBPF, like the byte code I mentioned, would
simplify the verifier in that case.
We have some type specific constructs already. For example we know
certain helpers will return a max value, etc. and can avoid extra
bounds checks. If we can safely make more assumptions based on
the types of the program and/or types of variables lets do it. I'm
all in for optimizations in the verifier side.
If you have specific complexity problems we can work to solve those
either in the clang backend or in verifier itself. Solving these
will help all use cases so again happy to help here.
quoted
quoted
quoted
quoted
quoted
quoted
quoted
Hi Jiri,
Yes, the intent is to replace flow dissector with a parser
that is
quoted
quoted
quoted
quoted
quoted
quoted
quoted
more extensible, more manageable and can be accelerated in
hardware
quoted
quoted
quoted
quoted
quoted
quoted
quoted
(good luck trying to HW accelerate flow dissector as is ;-)
Can we drop hyperbole? There are several examples of hardware
that
quoted
quoted
quoted
quoted
quoted
quoted
offload (a subset of) flower. That the current kernel
implementation has
quoted
quoted
quoted
quoted
quoted
quoted
the properties you describe is pretty much irrelevant for
current hw
quoted
quoted
quoted
quoted
quoted
quoted
offload use-cases.
Simon,
"current hw offload use-cases" is the problem; these models
offer no
quoted
quoted
quoted
quoted
quoted
extensibility. For instance, if a new protocol appears or a user
wants
quoted
quoted
quoted
quoted
quoted
to support their own custom protocol in things like tc-flower
there is
quoted
quoted
quoted
quoted
quoted
no feasible way to do this. Unfortunately, as of today it seems,
we
quoted
quoted
quoted
quoted
quoted
are still bound by the marketing department at hardware vendors
that
quoted
quoted
quoted
quoted
quoted
pick and choose the protocols that they think their customers
want and
quoted
quoted
quoted
quoted
quoted
are willing to invest in-- we need to get past this once and for
all!
quoted
quoted
quoted
quoted
quoted
IMO, what we need is a common way to extend the kernel, tc, and
other
quoted
quoted
quoted
quoted
quoted
applications for new protocols and features, but also be able to
apply
quoted
quoted
quoted
quoted
quoted
that method to extend to the hardware which is _offloading_
kernel
quoted
quoted
quoted
quoted
quoted
functionality which in this case is flow dissector. The
technology is
quoted
quoted
quoted
quoted
quoted
there to do this as programmable NICs for instance are the rage,
but
quoted
quoted
quoted
quoted
quoted
we do need to create common APIs to be able to do that. Note this
isn't just tc, but a whole space of features; for instance, XDP
hints
quoted
quoted
quoted
quoted
quoted
is nice idea for the NIC to provide information about protocols
in a
quoted
quoted
quoted
quoted
quoted
packet, but unless/until there is a way to program the device to
pull
quoted
quoted
quoted
quoted
quoted
out arbitrary information that the user cares about like
something
quoted
quoted
quoted
quoted
quoted
from their custom protocol, then it's very limited utility...
Vendors have the ability to code up arbitrary hints today. They just
haven't open sourced it or made it widely available. I don't see how
Vendors can do this, but can I do this as a user? If I have a custom
protocol can I write some code for that without calling my vendor and
can I use that with tc-flower offload as easily as any other protocol?
I believe this is what is needed.
I agree it would be great for the vendors to expose this, but I don't
see how flower2 gets us there yet. Said vendors could give us the
tools needed now, but haven't.
quoted
quoted
a 'tc' interface would help with this. I suspect most hardware could
prepend hints or put other arbitrary data in the descriptor or
elsewhere.
quoted
quoted
The compelling reason to open source it is missing.
I'm not sure about that. It's not so much a question of the mechanisms
to convey the data (although being restricted to just a few bytes in a
receive descriptor for XDP Hints is an obvious limitation), the
We have multi-buffer support coming so we could use pages of data
for metadata if we wanted fairly easily once that lands. We could
even have hardware DMA the metadata into a page and just add that
to the frag list without any copying.
quoted
problem is the generation of the hints themself. For instance, suppose
I want to extract the QUIC connection identifier as an XDP Hint? To do
this we would need the device to be able to parse in UDP payload and
extract the connection identifier. I don't believe that capability is
widespread yet, vendors have parsers but generally they have not made
them user programmable. And even if they did do that, then we need a
common interface to program that functionality-- proprietary APIs are
right out due to risk of vendor lockin. I imagine the P4 advocates
think they are the solution, but then we have to absorb the
maintenance cost of yet another domain specific language and P4 isn't
part of the kernel so it really isn't offloading kernel functionality.
This is the gap the PANDA fills.
I don't think P4 or Panda should be in-kernel. The kernel has a BPF
parser that can do arbitrary protocol parsing today. I don't see
a reason to add another thing on the chance a hardware offload
might come around. Anyways P4/Panda can compile to the BPF parser
or flower if they want and do their DSL magic on top. And sure
we might want to improve the clang backends, the existing flower
classifier, and BPF verifier.
BPF knows about skbs and its attach points. If it can assume
bounds lengths or other simplifying constraints automatically
lets do it.
Without hardware support I can't see any advantage to flower2.
John,
Please look at patch log, there are number of problems that have come up
flow dissector over the years. Most of this is related to inherent
inflexibility, limitations, missing support for fairly basic protocols, and
there's a lot of information loss because of the fixed monolithic data
structures. I've said it many times: skb_flow_dissect is the function we
love to hate. Maybe it's arguable, bit I claim it's 2000 lines of spaghetti
code. I don't think there's anyone to blame for that, this was a
consequence of evolving very useful feature that isn't really amenable to
being written in sequence of imperative instructions (if you recall it used
to be even worse with something like 20 goto's scattered about that defied
any semblance of logical program flow :-) ).
OK, but if thats the goal then shouldn't this series target replacing the
flow_dissector code directly? I don't see any edits to ./net/core.
The equivalent code in PANDA is far simpler, extensible, and maintainable
and there are opportunities for context aware optimizations that achieve
higher performance (we'll post performance numbers showing that shortly).
It's also portable to different environments both SW and HW.
If so replace flow_dissector then I think and lets debate that.
My first question as a flow dissector replacement would be the BPF
flow dissector was intended to solve the generic parsing problem.
Why would Panda be better? My assumption here is that BPF should
solve the generic parsing problem, but as we noted isn't very
friendly to HW offload. So we jumped immediately into HW offload
space. If the problem is tc_flower is not flexible enough
couldn't we make tc_flower use the BPF dissector? That should
still allow tc flower to do its offload above the sw BPF dissector
to hardware just fine.
I guess my first level question is why did BPF flow dissector
program not solve the SW generic parsing problem. I read the commit
messages and didn't find the answer.
.John
Tom
quoted
even if we had support the amount of firmware logic going into taking
that parse graph into hardware makes me a bit nervous. I would
prefer to see a proper compiler that generates the microcode and
*CAM table entires necessary to implement said switch logic from
any DSL although I prefer P4. Sure that compiler wont be in
kernel most likely, but I want transparency and I want tools
to configure my software how I want it. Neither requries a
kernel shim.
quoted
quoted
Then the flwo is fairly straight forward the XDP program reads the
hints. Then if the rest of the stack needs this in the skb we have
the hash and skb extensions.
Again that's the backend mechanism, the frontend problem is the
generation of the hints data itself.
Sure, but my point is the mechanism exists today to generate
arbitrary hints and no vendor has stepped up and provide
code to do it. The only conclusion I can come up with is its
not valuable on their side to do it.
quoted
quoted
quoted
quoted
... the NIC could run a BPF program if its programmable to that
extent.
quoted
quoted
quoted
quoted
Simon,
True, but that implies that the NIC would just be running code in one
CPU instead of another-- i.e., that is doing offload and not
acceleration. Hardware parses are more likely to be very specialized
and might look something like a parameterized FSM that runs 10x
faster
quoted
quoted
quoted
than software in a CPU. In order to be able to accelerate, we need to
start with a parser representation that is more declarative than
Agree, but I don't see how configuration of this hardware makes sense
over 'tc'. This is likely to require compiler tools to generate the
microcode or *CAM entries running on the hardware. Having 'tc' run
a few link, add-header commands that can be converted into reconfigured
hardware (thats not just a CPU on a NIC) seems like we will be asking
a lot of firmware. Probably too much for my taste, fixing bugs in
firmware is going to be harder then if vendors just give us the
compiler tools to generate the parsing logic for their hardware.
Show me the hardware that can create new parse trees using flower2,
do they exist?
Working on it :-)
OK I think flower2 should wait for the hardware then.
quoted
quoted
If not the flow is like this,
0. build new parse graph and hardware logic using DSL (P4 or
otherwise)
quoted
quoted
1. apply blob output from 0 onto hardware
2. build out flower2 graph
3. flower2 populates hardware but hardware already got it from 0?
I'm missing the point here?
quoted
imperative. This is what PANDA provides, the user writes a parser in
a
quoted
quoted
quoted
declarative representation (but still in C). Given the front end
representation is declarative, we can compile that to a type of byte
code that is digestible to instantiate a reasonably programmable
hardware parser. This fits well with eBPF where the byte code is
domain specific instructions to eBPF, so when the eBPF program runs
they can be JIT compiled into CPU instructions for running on the
host, but they can be given to driver that can translate or JIT
compile the byte code into their hardware parser (coud JIT compile to
P4 backend for instance).
I'm not sure hardware exists that can or will take arbitrary 'tc'
commands and build a graph of a new protocols? Also we already have
a SW path for arbitrary flow parser with BPF so I see no reasons
we need to support yet another one. Even PANDA could produce BPF
codes for sofwtare and hardware codes to program the hardware so
why is this needed?
Hardware parsers are not general purpose CPUs, but specialized engines
that expect input in a declarative representation of an annotated
Of course.
quoted
parse graph that includes parameterized functions needed to determine
next protocol and length. If we compile PANDA parser, which is in
declarative representation, into eBPF then we've lost the structure
and essence of the problem so that we can't accelerate the parser in
specialized hardware. The alternative I'm suggesting is to extend eBPF
to include the declarative representation of the parser. So when the
program is downloaded to the kernel it can run in CPU by compiling to
host ISA, the kernel can then give the program to the device that in
turn can instantiate it in the hardware.
This feels like multiple levels of abstraction for no reason.
Why not just open up the device and give it the declaration directly.
I know some people disagree, but why are we creating all this code
when the entire purpose can be done by simply passing the info to
the hardware directly. I see no reason BPF should pick up extra
complexity just so some hardware device can take it and convert
it out of BPF and run some annotations.
quoted
A critical aspect of this method also eliminates any sort of side
channel or kernel bypass to program the device, which means the device
programming is visible to the kernel hence there is no ambiguity as to
what the device is actually doing (this lack of transparency is why
protocol specific offloads like LRO have failed to get traction). This
visibility is necessary if we are ever to go beyond just getting a few
disposable hints from the device and are getting actual operational
data for the stack to consume (like accelerated TXDP where we can jump
straight to TCP receive processing routine from the driver because the
device was able to perform all the stateless TCP and IP processing
before giving the packet the host). Generally, I view this direction
as extending the boundary of the stack into the device.
Agree we need visibility into what the hardware is doing if we want
complex offloads. But, I don't see why we need to complicated the
software stacks to support the offload.
Anyways IMO we need hardware support to make heads or tails of this.
quoted
Tom
quoted
Also yes I saw the netdevconf but still missed the point Sorry.
Thanks
.John
My only concern is that is there any way to reuse flower
code instead
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
of duplicating most of them? Especially when you
specifically mentioned
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
flower2 has the same user-space syntax as flower, this
makes code
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
reusing more reasonable.
Exactly. I believe it is wrong to introduce new classifier
which would
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
basically behave exacly the same as flower, only has
different parser
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
implementation under the hood.
Could you please explore the possibility to replace
flow_dissector by
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
your dissector optionally at first (kernel config for
example)? And I'm
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
not talking only about flower, but about the rest of the
flow_dissector
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
users too.
+1
Does the existing BPF flow dissector not work for some reason? If its
purely
quoted
quoted
a hardware mapping problem, couple questions below.
Hi John,
eBPF in its current form is un-acceleratable is a primary problem,
We have no disagreement here. Offloading a general purpose instruction
set into an architecture (switch, etc.) that looks nothing like this
is a losing game.
quoted
however an eBPF flow dissector would still have the same issues in
complexity and manageability that the kernel flow dissector. PANDA
PANDA is a DSL and runtime, similarly P4 is another DSL and runtime
environment. P4 can compile to BPF, PANDA can as well. I don't think
PANDA or P4 belong in kernel. We can argue about preferred DSLs but
I don't think that is a @netdev @bpf concern.
quoted
pretty much can address that since the same source code for flow
dissector used in tc-flower could be compiled into eBPF and used with
TC (that gives extensibility at some incremental performance
degradation). There is an interesting caveat there in that to satisfy
verifier we had to break up the parser to avoid being flagged for
complexity. The parser code however already started with a restricted
structure that enforces rules that prevent the issues the verified is
checking for, when we get to verifier though it doesn't have any that
context and hence views it as general purpose code. Introducing domain
specific constructs into eBPF, like the byte code I mentioned, would
simplify the verifier in that case.
We have some type specific constructs already. For example we know
certain helpers will return a max value, etc. and can avoid extra
bounds checks. If we can safely make more assumptions based on
the types of the program and/or types of variables lets do it. I'm
all in for optimizations in the verifier side.
If you have specific complexity problems we can work to solve those
either in the clang backend or in verifier itself. Solving these
will help all use cases so again happy to help here.
quoted
quoted
quoted
quoted
quoted
quoted
quoted
Hi Jiri,
Yes, the intent is to replace flow dissector with a parser
that is
quoted
quoted
quoted
quoted
quoted
quoted
quoted
more extensible, more manageable and can be accelerated in
hardware
quoted
quoted
quoted
quoted
quoted
quoted
quoted
(good luck trying to HW accelerate flow dissector as is ;-)
Can we drop hyperbole? There are several examples of hardware
that
quoted
quoted
quoted
quoted
quoted
quoted
offload (a subset of) flower. That the current kernel
implementation has
quoted
quoted
quoted
quoted
quoted
quoted
the properties you describe is pretty much irrelevant for
current hw
quoted
quoted
quoted
quoted
quoted
quoted
offload use-cases.
Simon,
"current hw offload use-cases" is the problem; these models
offer no
quoted
quoted
quoted
quoted
quoted
extensibility. For instance, if a new protocol appears or a user
wants
quoted
quoted
quoted
quoted
quoted
to support their own custom protocol in things like tc-flower
there is
quoted
quoted
quoted
quoted
quoted
no feasible way to do this. Unfortunately, as of today it seems,
we
quoted
quoted
quoted
quoted
quoted
are still bound by the marketing department at hardware vendors
that
quoted
quoted
quoted
quoted
quoted
pick and choose the protocols that they think their customers
want and
quoted
quoted
quoted
quoted
quoted
are willing to invest in-- we need to get past this once and for
all!
quoted
quoted
quoted
quoted
quoted
IMO, what we need is a common way to extend the kernel, tc, and
other
quoted
quoted
quoted
quoted
quoted
applications for new protocols and features, but also be able to
apply
quoted
quoted
quoted
quoted
quoted
that method to extend to the hardware which is _offloading_
kernel
quoted
quoted
quoted
quoted
quoted
functionality which in this case is flow dissector. The
technology is
quoted
quoted
quoted
quoted
quoted
there to do this as programmable NICs for instance are the rage,
but
quoted
quoted
quoted
quoted
quoted
we do need to create common APIs to be able to do that. Note this
isn't just tc, but a whole space of features; for instance, XDP
hints
quoted
quoted
quoted
quoted
quoted
is nice idea for the NIC to provide information about protocols
in a
quoted
quoted
quoted
quoted
quoted
packet, but unless/until there is a way to program the device to
pull
quoted
quoted
quoted
quoted
quoted
out arbitrary information that the user cares about like
something
quoted
quoted
quoted
quoted
quoted
from their custom protocol, then it's very limited utility...
Vendors have the ability to code up arbitrary hints today. They just
haven't open sourced it or made it widely available. I don't see how
Vendors can do this, but can I do this as a user? If I have a custom
protocol can I write some code for that without calling my vendor and
can I use that with tc-flower offload as easily as any other protocol?
I believe this is what is needed.
I agree it would be great for the vendors to expose this, but I don't
see how flower2 gets us there yet. Said vendors could give us the
tools needed now, but haven't.
quoted
quoted
a 'tc' interface would help with this. I suspect most hardware could
prepend hints or put other arbitrary data in the descriptor or
elsewhere.
quoted
quoted
The compelling reason to open source it is missing.
I'm not sure about that. It's not so much a question of the mechanisms
to convey the data (although being restricted to just a few bytes in a
receive descriptor for XDP Hints is an obvious limitation), the
We have multi-buffer support coming so we could use pages of data
for metadata if we wanted fairly easily once that lands. We could
even have hardware DMA the metadata into a page and just add that
to the frag list without any copying.
quoted
problem is the generation of the hints themself. For instance, suppose
I want to extract the QUIC connection identifier as an XDP Hint? To do
this we would need the device to be able to parse in UDP payload and
extract the connection identifier. I don't believe that capability is
widespread yet, vendors have parsers but generally they have not made
them user programmable. And even if they did do that, then we need a
common interface to program that functionality-- proprietary APIs are
right out due to risk of vendor lockin. I imagine the P4 advocates
think they are the solution, but then we have to absorb the
maintenance cost of yet another domain specific language and P4 isn't
part of the kernel so it really isn't offloading kernel functionality.
This is the gap the PANDA fills.
I don't think P4 or Panda should be in-kernel. The kernel has a BPF
parser that can do arbitrary protocol parsing today. I don't see
a reason to add another thing on the chance a hardware offload
might come around. Anyways P4/Panda can compile to the BPF parser
or flower if they want and do their DSL magic on top. And sure
we might want to improve the clang backends, the existing flower
classifier, and BPF verifier.
BPF knows about skbs and its attach points. If it can assume
bounds lengths or other simplifying constraints automatically
lets do it.
Without hardware support I can't see any advantage to flower2.
John,
Please look at patch log, there are number of problems that have come up
flow dissector over the years. Most of this is related to inherent
inflexibility, limitations, missing support for fairly basic protocols, and
there's a lot of information loss because of the fixed monolithic data
structures. I've said it many times: skb_flow_dissect is the function we
love to hate. Maybe it's arguable, bit I claim it's 2000 lines of spaghetti
code. I don't think there's anyone to blame for that, this was a
consequence of evolving very useful feature that isn't really amenable to
being written in sequence of imperative instructions (if you recall it used
to be even worse with something like 20 goto's scattered about that defied
any semblance of logical program flow :-) ).
OK, but if thats the goal then shouldn't this series target replacing the
flow_dissector code directly? I don't see any edits to ./net/core.
John,
Well this is just an RFC ;-). We'll follow with the patch series that
more directly illustrates replacing kernel flow dissector.
quoted
The equivalent code in PANDA is far simpler, extensible, and maintainable
and there are opportunities for context aware optimizations that achieve
higher performance (we'll post performance numbers showing that shortly).
It's also portable to different environments both SW and HW.
If so replace flow_dissector then I think and lets debate that.
My first question as a flow dissector replacement would be the BPF
flow dissector was intended to solve the generic parsing problem.
Why would Panda be better?
To be clear PANDA is a programming model for the datapath not an
execution environment-- I view it as complementary to kernel, eBPF,
DPDK, etc. Similar to how we are suggesting to replace the kernel flow
dissector with a PANDA derived flow dissector, we can implement a BPF
flow dissector in PANDA as well. This should be as simple as
recompiling the parser we're using in this patch for eBPF (we'll do
that experiment). As I mentioned the immediate benefits are a
simplified code base and improved program structure in the source
source. With a context aware optimizing compiler we can output C code
that has higher performance than the equivalent handwritten imperative
code. The longer term benefits are that the same exact parser code is
now accleratable and a more user-friendly interface for users adding
their own custom protocols to generic tc-flower.
My assumption here is that BPF should
solve the generic parsing problem, but as we noted isn't very
friendly to HW offload.
Right, not friendly to HW offload but that's also correlated to not
being very friendly programmability in general.
So we jumped immediately into HW offload
space. If the problem is tc_flower is not flexible enough
couldn't we make tc_flower use the BPF dissector?
Yes, in the software model we would use a BPF dissector, I'm just
saying that that for generic tc-flower we would derive that from PANDA
which is going to be better performance and easier to make the changes
in TC flower application for dynamic CLI to use PANDA parser. Without
any additional considerations, all we're really doing is rewriting the
BPF and kernel flow dissector in a simpler model but retaining the
same functionality (although there is a nice advantage that both
instances can be derived from the same code base).
That should
still allow tc flower to do its offload above the sw BPF dissector
to hardware just fine.
Right but we need something to offload, and it really has to be
identical functionality to what kernel does. While we can't offload an
eBPF flow dissector, we can offload a flow dissector which starts from
an expression in PANDA and is also the basis for an eBPF one kernel
uses.
I guess my first level question is why did BPF flow dissector
program not solve the SW generic parsing problem. I read the commit
messages and didn't find the answer.
I do think eBPF is key to providing a generic software parser, doing
an eBPF parser in PANDA has the advantages I mentioned. This does lead
to a more profound question with regards to the kernel flow dissector
we have today in the kernel: can the flow dissector be completely
removed and replaced with just an eBPF hook that is populated with a
default parser program as part of kernel compilation? I believe the
major reason to have a kernel flow dissector instead of a default eBPF
flow dissector is for performance, presumably native kernel code is
faster than eBPF code in kernel. But if the eBPF code is actually
faster, like in PANDA where we can optimize the code with context,
then the performance issue might be moot. So now, instead of eBPF
being used as an optional hook, it's actively replacing some core
kernel functionality from boot (hence we're starting to rewrite the
kernel itself in eBPF which is why I call it profound). That is
potentially a big win for extensibility, flexibility, and
accelerability also. I think this is something we should explore, but
maybe take an incremental approach to get there by first moving the
kernel flow dissector to a simpler model.
Tom
.John
quoted
Tom
quoted
even if we had support the amount of firmware logic going into taking
that parse graph into hardware makes me a bit nervous. I would
prefer to see a proper compiler that generates the microcode and
*CAM table entires necessary to implement said switch logic from
any DSL although I prefer P4. Sure that compiler wont be in
kernel most likely, but I want transparency and I want tools
to configure my software how I want it. Neither requries a
kernel shim.
quoted
quoted
Then the flwo is fairly straight forward the XDP program reads the
hints. Then if the rest of the stack needs this in the skb we have
the hash and skb extensions.
Again that's the backend mechanism, the frontend problem is the
generation of the hints data itself.
Sure, but my point is the mechanism exists today to generate
arbitrary hints and no vendor has stepped up and provide
code to do it. The only conclusion I can come up with is its
not valuable on their side to do it.
quoted
quoted
quoted
quoted
... the NIC could run a BPF program if its programmable to that
extent.
quoted
quoted
quoted
quoted
Simon,
True, but that implies that the NIC would just be running code in one
CPU instead of another-- i.e., that is doing offload and not
acceleration. Hardware parses are more likely to be very specialized
and might look something like a parameterized FSM that runs 10x
faster
quoted
quoted
quoted
than software in a CPU. In order to be able to accelerate, we need to
start with a parser representation that is more declarative than
Agree, but I don't see how configuration of this hardware makes sense
over 'tc'. This is likely to require compiler tools to generate the
microcode or *CAM entries running on the hardware. Having 'tc' run
a few link, add-header commands that can be converted into reconfigured
hardware (thats not just a CPU on a NIC) seems like we will be asking
a lot of firmware. Probably too much for my taste, fixing bugs in
firmware is going to be harder then if vendors just give us the
compiler tools to generate the parsing logic for their hardware.
Show me the hardware that can create new parse trees using flower2,
do they exist?
Working on it :-)
OK I think flower2 should wait for the hardware then.
quoted
quoted
If not the flow is like this,
0. build new parse graph and hardware logic using DSL (P4 or
otherwise)
quoted
quoted
1. apply blob output from 0 onto hardware
2. build out flower2 graph
3. flower2 populates hardware but hardware already got it from 0?
I'm missing the point here?
quoted
imperative. This is what PANDA provides, the user writes a parser in
a
quoted
quoted
quoted
declarative representation (but still in C). Given the front end
representation is declarative, we can compile that to a type of byte
code that is digestible to instantiate a reasonably programmable
hardware parser. This fits well with eBPF where the byte code is
domain specific instructions to eBPF, so when the eBPF program runs
they can be JIT compiled into CPU instructions for running on the
host, but they can be given to driver that can translate or JIT
compile the byte code into their hardware parser (coud JIT compile to
P4 backend for instance).
I'm not sure hardware exists that can or will take arbitrary 'tc'
commands and build a graph of a new protocols? Also we already have
a SW path for arbitrary flow parser with BPF so I see no reasons
we need to support yet another one. Even PANDA could produce BPF
codes for sofwtare and hardware codes to program the hardware so
why is this needed?
Hardware parsers are not general purpose CPUs, but specialized engines
that expect input in a declarative representation of an annotated
Of course.
quoted
parse graph that includes parameterized functions needed to determine
next protocol and length. If we compile PANDA parser, which is in
declarative representation, into eBPF then we've lost the structure
and essence of the problem so that we can't accelerate the parser in
specialized hardware. The alternative I'm suggesting is to extend eBPF
to include the declarative representation of the parser. So when the
program is downloaded to the kernel it can run in CPU by compiling to
host ISA, the kernel can then give the program to the device that in
turn can instantiate it in the hardware.
This feels like multiple levels of abstraction for no reason.
Why not just open up the device and give it the declaration directly.
I know some people disagree, but why are we creating all this code
when the entire purpose can be done by simply passing the info to
the hardware directly. I see no reason BPF should pick up extra
complexity just so some hardware device can take it and convert
it out of BPF and run some annotations.
quoted
A critical aspect of this method also eliminates any sort of side
channel or kernel bypass to program the device, which means the device
programming is visible to the kernel hence there is no ambiguity as to
what the device is actually doing (this lack of transparency is why
protocol specific offloads like LRO have failed to get traction). This
visibility is necessary if we are ever to go beyond just getting a few
disposable hints from the device and are getting actual operational
data for the stack to consume (like accelerated TXDP where we can jump
straight to TCP receive processing routine from the driver because the
device was able to perform all the stateless TCP and IP processing
before giving the packet the host). Generally, I view this direction
as extending the boundary of the stack into the device.
Agree we need visibility into what the hardware is doing if we want
complex offloads. But, I don't see why we need to complicated the
software stacks to support the offload.
Anyways IMO we need hardware support to make heads or tails of this.
quoted
Tom
quoted
Also yes I saw the netdevconf but still missed the point Sorry.
Thanks
.John
From: Jamal Hadi Salim <jhs@mojatatu.com> Date: 2021-09-23 13:26:44
Geez, I missed all the fun ;->
On 2021-09-22 11:25 p.m., John Fastabend wrote:
Tom Herbert wrote:
quoted
On Wed, Sep 22, 2021, 6:29 PM John Fastabend [off-list ref]
wrote:
[..]
quoted
John,
Please look at patch log, there are number of problems that have come up
flow dissector over the years. Most of this is related to inherent
inflexibility, limitations, missing support for fairly basic protocols, and
there's a lot of information loss because of the fixed monolithic data
structures. I've said it many times: skb_flow_dissect is the function we
love to hate. Maybe it's arguable, bit I claim it's 2000 lines of spaghetti
code. I don't think there's anyone to blame for that, this was a
consequence of evolving very useful feature that isn't really amenable to
being written in sequence of imperative instructions (if you recall it used
to be even worse with something like 20 goto's scattered about that defied
any semblance of logical program flow :-) ).
OK, but if thats the goal then shouldn't this series target replacing the
flow_dissector code directly? I don't see any edits to ./net/core.
Agreed, replacement of flow dissector should be a focus. Jiri's
suggestion of a followup patch which shows how the rest of the consumers
of flow dissector could be made to use PANDA is a good idea.
IMO (correct me if i am wrong Tom), flower2 was merely intended to
illustrate how one would use PANDA i.e there are already two patches
of which the first one is essentially PANDA...
IOW, it is just flower but with flow dissector replaced by PANDA.
quoted
The equivalent code in PANDA is far simpler, extensible, and maintainable
and there are opportunities for context aware optimizations that achieve
higher performance (we'll post performance numbers showing that shortly).
It's also portable to different environments both SW and HW.
If so replace flow_dissector then I think and lets debate that.
My first question as a flow dissector replacement would be the BPF
flow dissector was intended to solve the generic parsing problem.
Why would Panda be better? My assumption here is that BPF should
solve the generic parsing problem, but as we noted isn't very
friendly to HW offload. So we jumped immediately into HW offload
space. If the problem is tc_flower is not flexible enough
couldn't we make tc_flower use the BPF dissector? That should
still allow tc flower to do its offload above the sw BPF dissector
to hardware just fine.
I guess my first level question is why did BPF flow dissector
program not solve the SW generic parsing problem. I read the commit
messages and didn't find the answer.
Sorry, you cant replace/flowdissector/BPF such that flower can
consume it;-> You are going to face a huge path explosion with the
verifier due to the required branching and then resort to all
kinds of speacial-cased acrobatics.
See some samples of XDP code going from trying to parse basic TCP
options to resorting to tricking the verifier.
For shits and giggles, as they say in Eastern Canada, try to do
IPV6 full parsing with BPF (and handle all the variable length
fields).
Generally:
BPF is good for specific smaller parsing tasks; the ebpf flow dissector
hook should be trivial to add to PANDA. And despite PANDA being able
to generate EBPF - I would still say it depends on the depth of the
parse tree to be sensible to use eBPF.
Earlier in the thread you said a couple of things that caught my
attention:
> I don't think P4 or Panda should be in-kernel. The kernel has a BPF
> parser that can do arbitrary protocol parsing today. I don't see
> a reason to add another thing on the chance a hardware offload
> might come around. Anyways P4/Panda can compile to the BPF parser
> or flower if they want and do their DSL magic on top. And sure
> we might want to improve the clang backends, the existing flower
> classifier, and BPF verifier.
>
>
> Vendors have the ability to code up arbitrary hints today. They just
> haven't open sourced it or made it widely available. I don't see how
> a 'tc' interface would help with this. I suspect most hardware could
> prepend hints or put other arbitrary data in the descriptor or elsewhere.
> The compelling reason to open source it is missing.
Please, please _lets not_ encourage vendors to continue
keep things proprietary!
Statements like "I don't think P4 or Panda should be in-kernel..."
are just too strong.
Instead lets focus on how we can make P4 and other hardware offloads
work in conjunction with the kernel (instead of totally bypassing
it which is what vendors are doing enmasse already). There are
billions of $ invested in these ASICs and lets welcome them into
our world. It serves and helps grow the Linux community better.
The efforts of switchdev and tc offloading have proven it is possible.
Vendors (and i am going to call out Broadcom on the switching side here)
are not partaking because they see it as an economical advantage not to
partake.
We have learnt a lot technically since switchdev/tc offloads happened.
So it is doable.
The first rule is: In order to get h/w offload to work lets also have
digitally equivalent implementation in s/w.
cheers,
jamal
From: John Fastabend <john.fastabend@gmail.com> Date: 2021-09-24 03:55:34
Jamal Hadi Salim wrote:
Geez, I missed all the fun ;->
On 2021-09-22 11:25 p.m., John Fastabend wrote:
quoted
Tom Herbert wrote:
quoted
On Wed, Sep 22, 2021, 6:29 PM John Fastabend [off-list ref]
wrote:
[..]
quoted
quoted
John,
Please look at patch log, there are number of problems that have come up
flow dissector over the years. Most of this is related to inherent
inflexibility, limitations, missing support for fairly basic protocols, and
there's a lot of information loss because of the fixed monolithic data
structures. I've said it many times: skb_flow_dissect is the function we
love to hate. Maybe it's arguable, bit I claim it's 2000 lines of spaghetti
code. I don't think there's anyone to blame for that, this was a
consequence of evolving very useful feature that isn't really amenable to
being written in sequence of imperative instructions (if you recall it used
to be even worse with something like 20 goto's scattered about that defied
any semblance of logical program flow :-) ).
OK, but if thats the goal then shouldn't this series target replacing the
flow_dissector code directly? I don't see any edits to ./net/core.
Agreed, replacement of flow dissector should be a focus. Jiri's
suggestion of a followup patch which shows how the rest of the consumers
of flow dissector could be made to use PANDA is a good idea.
I'de almost propose starting with flow_dissector.c first so we see that the
./net/core user for the SW only case looks good. Although I like the idea
of doing it all in BPF directly so could take a crack at that as well. Then
compare them.
IMO (correct me if i am wrong Tom), flower2 was merely intended to
illustrate how one would use PANDA i.e there are already two patches
of which the first one is essentially PANDA...
IOW, it is just flower but with flow dissector replaced by PANDA.
quoted
quoted
The equivalent code in PANDA is far simpler, extensible, and maintainable
and there are opportunities for context aware optimizations that achieve
higher performance (we'll post performance numbers showing that shortly).
It's also portable to different environments both SW and HW.
If so replace flow_dissector then I think and lets debate that.
My first question as a flow dissector replacement would be the BPF
flow dissector was intended to solve the generic parsing problem.
quoted
Why would Panda be better? My assumption here is that BPF should
solve the generic parsing problem, but as we noted isn't very
friendly to HW offload. So we jumped immediately into HW offload
space. If the problem is tc_flower is not flexible enough
couldn't we make tc_flower use the BPF dissector? That should
still allow tc flower to do its offload above the sw BPF dissector
to hardware just fine.
I guess my first level question is why did BPF flow dissector
program not solve the SW generic parsing problem. I read the commit
messages and didn't find the answer.
Sorry, you cant replace/flowdissector/BPF such that flower can
consume it;-> You are going to face a huge path explosion with the
verifier due to the required branching and then resort to all
kinds of speacial-cased acrobatics.
See some samples of XDP code going from trying to parse basic TCP
options to resorting to tricking the verifier.
For shits and giggles, as they say in Eastern Canada, try to do
IPV6 full parsing with BPF (and handle all the variable length
fields).
We parse TLVs already and it works just fine. It requires some
careful consideration and clang does some dumb things here and
there, but it is doable. Sure verifier could maybe be improved
around a few cases and C frontend gets in the way sometimes,
but PANDA or P4 or other DSL could rewrite in LLVM-IR directly
to get the correct output.
Generally:
BPF is good for specific smaller parsing tasks; the ebpf flow dissector
hook should be trivial to add to PANDA. And despite PANDA being able
to generate EBPF - I would still say it depends on the depth of the
parse tree to be sensible to use eBPF.
Going to disagree. I'm fairly confident we could write a BPF
program to do the flow disection. Anyways we can always improve
the verifier as needed and this helps lots of things not
just this bit. Also flow dissector will be loaded once at early
boot most likely so we can allow it to take a bit longer or
pre-verify it. Just ideas.
Earlier in the thread you said a couple of things that caught my
attention:
> I don't think P4 or Panda should be in-kernel. The kernel has a BPF
> parser that can do arbitrary protocol parsing today. I don't see
> a reason to add another thing on the chance a hardware offload
> might come around. Anyways P4/Panda can compile to the BPF parser
> or flower if they want and do their DSL magic on top. And sure
> we might want to improve the clang backends, the existing flower
> classifier, and BPF verifier.
>
>
> Vendors have the ability to code up arbitrary hints today. They just
> haven't open sourced it or made it widely available. I don't see how
> a 'tc' interface would help with this. I suspect most hardware could
> prepend hints or put other arbitrary data in the descriptor or elsewhere.
> The compelling reason to open source it is missing.
Please, please _lets not_ encourage vendors to continue
keep things proprietary!
Fair enough. Some frustration leaking in from my side knowing
the hardware has been around for years and we've seen multiple
proposals but only limited hardware backing. Tom mentioned
he was working on the hardware angle so perhaps its close.
Statements like "I don't think P4 or Panda should be in-kernel..."
are just too strong.
Where I wanted to go with this is P4 and Panda are DSLs in my
mind. I think we should keep the kernel non-specific to any
one DSL. We should have a low level generic way to add them
to the kernel, I think this is BPF. Then we let users pick
whatever DSL they like and/or make up their own DSL.
Is the counter-argument that Panda is not a DSL, but rather
a low-level parser builder pattern.
Instead lets focus on how we can make P4 and other hardware offloads
work in conjunction with the kernel (instead of totally bypassing
it which is what vendors are doing enmasse already). There are
billions of $ invested in these ASICs and lets welcome them into
our world. It serves and helps grow the Linux community better.
The efforts of switchdev and tc offloading have proven it is possible.
Vendors (and i am going to call out Broadcom on the switching side here)
are not partaking because they see it as an economical advantage not to
partake.
We have learnt a lot technically since switchdev/tc offloads happened.
So it is doable.
The first rule is: In order to get h/w offload to work lets also have
digitally equivalent implementation in s/w.
But there is a cost to this its yet another bit of software to
maintain and review and so on. I'm arguing we already have a generic
way to implement h/w equivalence and its BPF. So instead of inventing
another method to do software we can improve the BPF layer. If we need
to build semantics over it that look consumable to hw so be it.
Also the other core question I still don't understand is how a
piece of hardware could consume a parse graph piece-meal through
an interface like proposed in flower2 and generate an arbitrary
parse graph? On the fly none the less. That feels like some very
powerful firmware to me.
And I would prefer open source userspace code (non-kernel) to
deep magic in firmware. At least then I can see it, patch it,
fix it, etc.
Last thing, I'll point out I got back deep into the hardware debate.
I'm still not convinced its the right thing to rip out the flow
dissector piece and replace it with Panda.
Thanks!
John
From: Tom Herbert <hidden> Date: 2021-09-24 16:22:07
On Thu, Sep 23, 2021 at 8:55 PM John Fastabend [off-list ref] wrote:
Jamal Hadi Salim wrote:
quoted
Geez, I missed all the fun ;->
On 2021-09-22 11:25 p.m., John Fastabend wrote:
quoted
Tom Herbert wrote:
quoted
On Wed, Sep 22, 2021, 6:29 PM John Fastabend [off-list ref]
wrote:
[..]
quoted
quoted
John,
Please look at patch log, there are number of problems that have come up
flow dissector over the years. Most of this is related to inherent
inflexibility, limitations, missing support for fairly basic protocols, and
there's a lot of information loss because of the fixed monolithic data
structures. I've said it many times: skb_flow_dissect is the function we
love to hate. Maybe it's arguable, bit I claim it's 2000 lines of spaghetti
code. I don't think there's anyone to blame for that, this was a
consequence of evolving very useful feature that isn't really amenable to
being written in sequence of imperative instructions (if you recall it used
to be even worse with something like 20 goto's scattered about that defied
any semblance of logical program flow :-) ).
OK, but if thats the goal then shouldn't this series target replacing the
flow_dissector code directly? I don't see any edits to ./net/core.
Agreed, replacement of flow dissector should be a focus. Jiri's
suggestion of a followup patch which shows how the rest of the consumers
of flow dissector could be made to use PANDA is a good idea.
I'de almost propose starting with flow_dissector.c first so we see that the
./net/core user for the SW only case looks good. Although I like the idea
of doing it all in BPF directly so could take a crack at that as well. Then
compare them.
That's an interesting idea and the intent, but note that one of the
reasons we are able to outperform flow dissectors is that
flow_dissector is parameterized to be generalized whereas PANDA Parser
would provide a customized instance for each use case. This is
especially evident in the bits to configure what data fields extracts,
in PANDA the extraction is explicit so a whole bunch of conditionals
in the datapath are eliminated. Replacing flow dissectors might look
more like creating a parser instance for each caller instead of
calling one function that tries to solve all problems.
quoted
IMO (correct me if i am wrong Tom), flower2 was merely intended to
illustrate how one would use PANDA i.e there are already two patches
of which the first one is essentially PANDA...
IOW, it is just flower but with flow dissector replaced by PANDA.
quoted
quoted
The equivalent code in PANDA is far simpler, extensible, and maintainable
and there are opportunities for context aware optimizations that achieve
higher performance (we'll post performance numbers showing that shortly).
It's also portable to different environments both SW and HW.
If so replace flow_dissector then I think and lets debate that.
My first question as a flow dissector replacement would be the BPF
flow dissector was intended to solve the generic parsing problem.
quoted
Why would Panda be better? My assumption here is that BPF should
solve the generic parsing problem, but as we noted isn't very
friendly to HW offload. So we jumped immediately into HW offload
space. If the problem is tc_flower is not flexible enough
couldn't we make tc_flower use the BPF dissector? That should
still allow tc flower to do its offload above the sw BPF dissector
to hardware just fine.
I guess my first level question is why did BPF flow dissector
program not solve the SW generic parsing problem. I read the commit
messages and didn't find the answer.
Sorry, you cant replace/flowdissector/BPF such that flower can
consume it;-> You are going to face a huge path explosion with the
verifier due to the required branching and then resort to all
kinds of speacial-cased acrobatics.
See some samples of XDP code going from trying to parse basic TCP
options to resorting to tricking the verifier.
For shits and giggles, as they say in Eastern Canada, try to do
IPV6 full parsing with BPF (and handle all the variable length
fields).
We parse TLVs already and it works just fine. It requires some
careful consideration and clang does some dumb things here and
there, but it is doable. Sure verifier could maybe be improved
around a few cases and C frontend gets in the way sometimes,
but PANDA or P4 or other DSL could rewrite in LLVM-IR directly
to get the correct output.
Currently the kernel flow dissector doesn't parse TLVs, for instance
Hop-by-Hop, DestOpts, IP options, and TCP options are just skipped.
TLVs are also the bane of router vendors since they despise
implementing protocols that require serialized processing over
combinatorial collection of elements. We need to get past this since
TLVs are a protocol extensibility which means they need to be a first
class citizen in a programmable parser API. (to be clear I'm not
saying anyone should add hardcordes TLV processing to
__skb_flow_dissect, it's already bloated enough!)
quoted
Generally:
BPF is good for specific smaller parsing tasks; the ebpf flow dissector
hook should be trivial to add to PANDA. And despite PANDA being able
to generate EBPF - I would still say it depends on the depth of the
parse tree to be sensible to use eBPF.
Going to disagree. I'm fairly confident we could write a BPF
program to do the flow disection. Anyways we can always improve
the verifier as needed and this helps lots of things not
just this bit. Also flow dissector will be loaded once at early
boot most likely so we can allow it to take a bit longer or
pre-verify it. Just ideas.
Yes, we already have a panda-compiler that converts PANDA-C in
well-optimized eBPF code. Per Jamal's point, that did require breaking
up the program into different tails calls. I believe once we hit four
layers of protocols we do a tail call and also do a tail call for each
instance of TLV processing. Note the
tools/testing/selftests/bpf/progs/bpf_flow.c has to deal with this
also and does this by statically making every L3 protocol into a tail
call (result is more tail calls than equivalent code PANDA).
quoted
Earlier in the thread you said a couple of things that caught my
attention:
> I don't think P4 or Panda should be in-kernel. The kernel has a BPF
> parser that can do arbitrary protocol parsing today. I don't see
> a reason to add another thing on the chance a hardware offload
> might come around. Anyways P4/Panda can compile to the BPF parser
> or flower if they want and do their DSL magic on top. And sure
> we might want to improve the clang backends, the existing flower
> classifier, and BPF verifier.
>
>
> Vendors have the ability to code up arbitrary hints today. They just
> haven't open sourced it or made it widely available. I don't see how
> a 'tc' interface would help with this. I suspect most hardware could
> prepend hints or put other arbitrary data in the descriptor or elsewhere.
> The compelling reason to open source it is missing.
Please, please _lets not_ encourage vendors to continue
keep things proprietary!
Fair enough. Some frustration leaking in from my side knowing
the hardware has been around for years and we've seen multiple
proposals but only limited hardware backing. Tom mentioned
he was working on the hardware angle so perhaps its close.
I share that frustration!
quoted
Statements like "I don't think P4 or Panda should be in-kernel..."
are just too strong.
Where I wanted to go with this is P4 and Panda are DSLs in my
mind. I think we should keep the kernel non-specific to any
one DSL. We should have a low level generic way to add them
to the kernel, I think this is BPF. Then we let users pick
whatever DSL they like and/or make up their own DSL.
Is the counter-argument that Panda is not a DSL, but rather
a low-level parser builder pattern.
Yes, PANDA is _not_ a DSL in the sense that it uses a new compiler,
tool chain, or skill sets to write a program all of which are required
for using P4. PANDA-C (just to make it look like the analogous CUDA-C
:-) ) is inherently C code that has a program structure for the
"low-level parser builder pattern".
quoted
Instead lets focus on how we can make P4 and other hardware offloads
work in conjunction with the kernel (instead of totally bypassing
it which is what vendors are doing enmasse already). There are
billions of $ invested in these ASICs and lets welcome them into
our world. It serves and helps grow the Linux community better.
The efforts of switchdev and tc offloading have proven it is possible.
Vendors (and i am going to call out Broadcom on the switching side here)
are not partaking because they see it as an economical advantage not to
partake.
We have learnt a lot technically since switchdev/tc offloads happened.
So it is doable.
The first rule is: In order to get h/w offload to work lets also have
digitally equivalent implementation in s/w.
But there is a cost to this its yet another bit of software to
maintain and review and so on. I'm arguing we already have a generic
way to implement h/w equivalence and its BPF. So instead of inventing
another method to do software we can improve the BPF layer. If we need
to build semantics over it that look consumable to hw so be it.
Also the other core question I still don't understand is how a
piece of hardware could consume a parse graph piece-meal through
an interface like proposed in flower2 and generate an arbitrary
parse graph? On the fly none the less. That feels like some very
powerful firmware to me.
And I would prefer open source userspace code (non-kernel) to
deep magic in firmware. At least then I can see it, patch it,
fix it, etc.
An instance of a parser is inherently a parse graph, so it follows
that the best representation for a parser is a declarative
representation of the parse graph. There was a watershed paper on this
by Nick McKeown and others in "Design principles for packet parsers".
A common declarative representation is then amenable to instantiation
in a hardware engine which is designed to consume that representation
(i.e. hardware parsers are basically programmable FSMs), and it's
equally straightforward to elicit a parser from a declarative
representation into imperative code for running in CPU. In this
regard, the only salient difference between P4 and PANDA-C is that the
declarative representation can be coded in PANDA as a graph data
structure in standard C, and in P4 the encoding is in an explicit
language construct.
Last thing, I'll point out I got back deep into the hardware debate.
I'm still not convinced its the right thing to rip out the flow
dissector piece and replace it with Panda.
Hardware acceleration is one goal, but there are merits in just
considering the software especially in regards to performance and
usability. We'll highlight those benefits in future patch series.
Tom
From: John Fastabend <john.fastabend@gmail.com> Date: 2021-09-24 19:14:23
Tom Herbert wrote:
On Thu, Sep 23, 2021 at 8:55 PM John Fastabend [off-list ref] wrote:
quoted
Jamal Hadi Salim wrote:
quoted
Geez, I missed all the fun ;->
On 2021-09-22 11:25 p.m., John Fastabend wrote:
quoted
Tom Herbert wrote:
quoted
On Wed, Sep 22, 2021, 6:29 PM John Fastabend [off-list ref]
wrote:
[..]
quoted
quoted
John,
Please look at patch log, there are number of problems that have come up
flow dissector over the years. Most of this is related to inherent
inflexibility, limitations, missing support for fairly basic protocols, and
there's a lot of information loss because of the fixed monolithic data
structures. I've said it many times: skb_flow_dissect is the function we
love to hate. Maybe it's arguable, bit I claim it's 2000 lines of spaghetti
code. I don't think there's anyone to blame for that, this was a
consequence of evolving very useful feature that isn't really amenable to
being written in sequence of imperative instructions (if you recall it used
to be even worse with something like 20 goto's scattered about that defied
any semblance of logical program flow :-) ).
OK, but if thats the goal then shouldn't this series target replacing the
flow_dissector code directly? I don't see any edits to ./net/core.
Agreed, replacement of flow dissector should be a focus. Jiri's
suggestion of a followup patch which shows how the rest of the consumers
of flow dissector could be made to use PANDA is a good idea.
I'de almost propose starting with flow_dissector.c first so we see that the
./net/core user for the SW only case looks good. Although I like the idea
of doing it all in BPF directly so could take a crack at that as well. Then
compare them.
That's an interesting idea and the intent, but note that one of the
reasons we are able to outperform flow dissectors is that
flow_dissector is parameterized to be generalized whereas PANDA Parser
would provide a customized instance for each use case. This is
especially evident in the bits to configure what data fields extracts,
in PANDA the extraction is explicit so a whole bunch of conditionals
in the datapath are eliminated. Replacing flow dissectors might look
more like creating a parser instance for each caller instead of
calling one function that tries to solve all problems.
Can you say a bit more on how a user would configure PANDA parser
from user side? For some reason I was under the impression that
users could push commands down to build the parser, but when I just
read the commit messages again I didn't see the details. Did I
make this up?
It feels like a good fit for BPF under the hood at least. User could
use Panda to build the BPF parser and then "load" it. The BPF parser
is then customized for each user and we already have the hook for
it in flow dissector and in tc.
This is the jump I can't make. I get that writing BPF can be
challenging so we want a parser language on top to help users.
But I'm missing the comparison between C based Panda parser vs
native BPF which we already have support for.
quoted
quoted
IMO (correct me if i am wrong Tom), flower2 was merely intended to
illustrate how one would use PANDA i.e there are already two patches
of which the first one is essentially PANDA...
IOW, it is just flower but with flow dissector replaced by PANDA.
quoted
quoted
The equivalent code in PANDA is far simpler, extensible, and maintainable
and there are opportunities for context aware optimizations that achieve
higher performance (we'll post performance numbers showing that shortly).
It's also portable to different environments both SW and HW.
If so replace flow_dissector then I think and lets debate that.
My first question as a flow dissector replacement would be the BPF
flow dissector was intended to solve the generic parsing problem.
quoted
Why would Panda be better? My assumption here is that BPF should
solve the generic parsing problem, but as we noted isn't very
friendly to HW offload. So we jumped immediately into HW offload
space. If the problem is tc_flower is not flexible enough
couldn't we make tc_flower use the BPF dissector? That should
still allow tc flower to do its offload above the sw BPF dissector
to hardware just fine.
I guess my first level question is why did BPF flow dissector
program not solve the SW generic parsing problem. I read the commit
messages and didn't find the answer.
Sorry, you cant replace/flowdissector/BPF such that flower can
consume it;-> You are going to face a huge path explosion with the
verifier due to the required branching and then resort to all
kinds of speacial-cased acrobatics.
See some samples of XDP code going from trying to parse basic TCP
options to resorting to tricking the verifier.
For shits and giggles, as they say in Eastern Canada, try to do
IPV6 full parsing with BPF (and handle all the variable length
fields).
We parse TLVs already and it works just fine. It requires some
careful consideration and clang does some dumb things here and
there, but it is doable. Sure verifier could maybe be improved
around a few cases and C frontend gets in the way sometimes,
but PANDA or P4 or other DSL could rewrite in LLVM-IR directly
to get the correct output.
Currently the kernel flow dissector doesn't parse TLVs, for instance
Hop-by-Hop, DestOpts, IP options, and TCP options are just skipped.
TLVs are also the bane of router vendors since they despise
implementing protocols that require serialized processing over
combinatorial collection of elements. We need to get past this since
TLVs are a protocol extensibility which means they need to be a first
class citizen in a programmable parser API. (to be clear I'm not
saying anyone should add hardcordes TLV processing to
__skb_flow_dissect, it's already bloated enough!)
No arguments from me here.
quoted
quoted
Generally:
BPF is good for specific smaller parsing tasks; the ebpf flow dissector
hook should be trivial to add to PANDA. And despite PANDA being able
to generate EBPF - I would still say it depends on the depth of the
parse tree to be sensible to use eBPF.
Going to disagree. I'm fairly confident we could write a BPF
program to do the flow disection. Anyways we can always improve
the verifier as needed and this helps lots of things not
just this bit. Also flow dissector will be loaded once at early
boot most likely so we can allow it to take a bit longer or
pre-verify it. Just ideas.
Yes, we already have a panda-compiler that converts PANDA-C in
well-optimized eBPF code. Per Jamal's point, that did require breaking
up the program into different tails calls. I believe once we hit four
layers of protocols we do a tail call and also do a tail call for each
instance of TLV processing. Note the
tools/testing/selftests/bpf/progs/bpf_flow.c has to deal with this
also and does this by statically making every L3 protocol into a tail
call (result is more tail calls than equivalent code PANDA).
Cool, so kernel has everything needed to run the eBPF parser why do
we also need the C parser in kernel? Performance wise they should
be comparable if not lets fix up the BPF side.
quoted
quoted
Earlier in the thread you said a couple of things that caught my
attention:
> I don't think P4 or Panda should be in-kernel. The kernel has a BPF
> parser that can do arbitrary protocol parsing today. I don't see
> a reason to add another thing on the chance a hardware offload
> might come around. Anyways P4/Panda can compile to the BPF parser
> or flower if they want and do their DSL magic on top. And sure
> we might want to improve the clang backends, the existing flower
> classifier, and BPF verifier.
>
>
> Vendors have the ability to code up arbitrary hints today. They just
> haven't open sourced it or made it widely available. I don't see how
> a 'tc' interface would help with this. I suspect most hardware could
> prepend hints or put other arbitrary data in the descriptor or elsewhere.
> The compelling reason to open source it is missing.
Please, please _lets not_ encourage vendors to continue
keep things proprietary!
Fair enough. Some frustration leaking in from my side knowing
the hardware has been around for years and we've seen multiple
proposals but only limited hardware backing. Tom mentioned
he was working on the hardware angle so perhaps its close.
I share that frustration!
quoted
quoted
Statements like "I don't think P4 or Panda should be in-kernel..."
are just too strong.
Where I wanted to go with this is P4 and Panda are DSLs in my
mind. I think we should keep the kernel non-specific to any
one DSL. We should have a low level generic way to add them
to the kernel, I think this is BPF. Then we let users pick
whatever DSL they like and/or make up their own DSL.
Is the counter-argument that Panda is not a DSL, but rather
a low-level parser builder pattern.
Yes, PANDA is _not_ a DSL in the sense that it uses a new compiler,
tool chain, or skill sets to write a program all of which are required
for using P4. PANDA-C (just to make it look like the analogous CUDA-C
:-) ) is inherently C code that has a program structure for the
"low-level parser builder pattern".
quoted
quoted
Instead lets focus on how we can make P4 and other hardware offloads
work in conjunction with the kernel (instead of totally bypassing
it which is what vendors are doing enmasse already). There are
billions of $ invested in these ASICs and lets welcome them into
our world. It serves and helps grow the Linux community better.
The efforts of switchdev and tc offloading have proven it is possible.
Vendors (and i am going to call out Broadcom on the switching side here)
are not partaking because they see it as an economical advantage not to
partake.
We have learnt a lot technically since switchdev/tc offloads happened.
So it is doable.
The first rule is: In order to get h/w offload to work lets also have
digitally equivalent implementation in s/w.
But there is a cost to this its yet another bit of software to
maintain and review and so on. I'm arguing we already have a generic
way to implement h/w equivalence and its BPF. So instead of inventing
another method to do software we can improve the BPF layer. If we need
to build semantics over it that look consumable to hw so be it.
Also the other core question I still don't understand is how a
piece of hardware could consume a parse graph piece-meal through
an interface like proposed in flower2 and generate an arbitrary
parse graph? On the fly none the less. That feels like some very
powerful firmware to me.
And I would prefer open source userspace code (non-kernel) to
deep magic in firmware. At least then I can see it, patch it,
fix it, etc.
An instance of a parser is inherently a parse graph, so it follows
that the best representation for a parser is a declarative
representation of the parse graph. There was a watershed paper on this
by Nick McKeown and others in "Design principles for packet parsers".
A common declarative representation is then amenable to instantiation
in a hardware engine which is designed to consume that representation
(i.e. hardware parsers are basically programmable FSMs), and it's
equally straightforward to elicit a parser from a declarative
representation into imperative code for running in CPU. In this
regard, the only salient difference between P4 and PANDA-C is that the
declarative representation can be coded in PANDA as a graph data
structure in standard C, and in P4 the encoding is in an explicit
language construct.
Sure agree with all of the above.
quoted
Last thing, I'll point out I got back deep into the hardware debate.
I'm still not convinced its the right thing to rip out the flow
dissector piece and replace it with Panda.
Hardware acceleration is one goal, but there are merits in just
considering the software especially in regards to performance and
usability. We'll highlight those benefits in future patch series.
So usability would be the PADNA-C representation. Good I'm all for
better user experience.
From performance side I think it should be almost the same between
your Panda-C runner and the BPF jitted code. If not I would want
to take a close look as to why not. Improving this BPF generated
code would help lots of software running today. For example
our XDP LB does parsing and we would love for it to be faster
if its possible.
I keep coming back to this. For software case use BPF the
infrastructure is there and we already have the hooks in
both flow dissector and 'tc' space. Now BPF is not the
best for offloading OK fine, can we just do this,
declarative-parse-graph --- sw case -> BPF
--- hw case -> hw-cmds
But, the problem I can't understand without more hardware details
is how the declarative parse graph makes the jump from some C
code into a new hw parser instantiation. IMO before we continue
with this RFC we really need to understand that piece. Otherwise
we have no way to see if these patches will work on hw and we
already have the sw case covered as best I can tell.
Also we would want at least multiple hardware vendors to
review it and agree its something that can work across
multiple hardware devices. FWIW I'm not a complete stranger
to how parsers run in the hardware and how they get encoded
even if my info is a few years out of date.
Thanks,
John
From: Jamal Hadi Salim <jhs@mojatatu.com> Date: 2021-09-26 15:54:44
On 2021-09-23 11:55 p.m., John Fastabend wrote:
Jamal Hadi Salim wrote:
[..]
quoted
Generally:
BPF is good for specific smaller parsing tasks; the ebpf flow dissector
hook should be trivial to add to PANDA. And despite PANDA being able
to generate EBPF - I would still say it depends on the depth of the
parse tree to be sensible to use eBPF.
Going to disagree. I'm fairly confident we could write a BPF
program to do the flow disection. Anyways we can always improve
the verifier as needed and this helps lots of things not
just this bit. Also flow dissector will be loaded once at early
boot most likely so we can allow it to take a bit longer or
pre-verify it. Just ideas.
With current status quo, it is challenging to achieve that
goal. Challenge is complexity and not so much instruction count.
In particular: If you include the parser as part of your eBPF
program you will reduce the amount of what the rest of your program
can do; depending how big the parse tree is, you may have no space
left at all for extra logic (even with tailcalls etc).
Changes to the eBPF core + verifier will certainly help - but
that remains to be seen. Hopefully the next patch set Felipe posts
would also have PANDA output for eBPF - then lets see what magic
you say can be added on top ;->
Note: I am not against ebpf, just where the fit is needs discussion.
For h/w interfacing, thats a totally different ballgame altogether
(tc is a good fit).
cheers,
jamal