On Mon, Jul 18, 2016 at 01:26:09PM +0000, Chandran, Sugesh wrote:

Hi Adrien,
Thank you for getting back on this.
Please find my comments below.

Hi Sugesh,

Same for me, removed again the parts we agree on.

[...]

quoted

quoted

quoted

quoted

quoted

quoted

[Sugesh] Another concern is the cost and time of installing
these rules in the hardware. Can we make these APIs time
bound(or at least an option

to

quoted

set the time limit to execute these APIs), so that Application
doesn’t have to wait so long when installing and deleting
flows

with

quoted

slow hardware/NIC. What do you think? Most of the datapath
flow

installations are

quoted

dynamic and triggered only when there is an ingress traffic.
Delay in flow insertion/deletion have unpredictable

consequences.

This API is (currently) aimed at the control path only, and must
indeed be assumed to be slow. Creating million of rules may take
quite long as it may involve syscalls and other time-consuming
synchronization things on the PMD side.

So currently there is no plan to have rules added from the data
path with time constraints. I think it would be implemented
through a different set of functions anyway.

I do not think adding time limits is practical, even specifying
in the API that creating a single flow rule must take less than
a maximum number of seconds in order to be effective is too much
of a constraint (applications that create all flows during init
may not care after

all).

quoted

quoted

You should consider in any case that modifying flow rules will
always be slower than receiving packets, there is no way around
that. Applications have to live with it and provide a software
fallback for incoming packets while managing flow rules.

Moreover, think about what happens when you hit the maximum

number

quoted

quoted

of flow rules and cannot create any more. Applications need to
implement some kind of fallback in their data path.

Offloading flows in HW is also only useful if they live much
longer than the time taken to create and delete them. Perhaps
applications may choose to do so after detecting long lived
flows such as TCP sessions.

You may have one separate control thread dedicated to manage
flows and keep your normal control thread unaffected by delays.
Several threads can even be dedicated, one per device.

[Sugesh] I agree that the flow insertion cannot be as fast as the
packet receiving rate.  From application point of view the problem
will be when hardware flow insertion takes longer than software
flow insertion. At least application has to know the cost of
inserting/deleting a rule in hardware beforehand. Otherwise how
application can choose the right flow candidate for hardware. My
point

here is application is expecting a deterministic behavior from a
classifier while inserting and deleting rules.

Understood, however it will be difficult to estimate, particularly
if a PMD must rearrange flow rules to make room for a new one due to
priority levels collision or some other HW-related reason. I mean,
spent time cannot be assumed to be constant, even PMDs cannot know
in advance because it also depends on the performance of the host CPU.

Such applications may find it easier to measure elapsed time for the
rules they create, make statistics and extrapolate from this
information for future rules. I do not think the PMD can help much here.

[Sugesh] From an application point of view this can be an issue.
Even there is a security concern when we program a short lived flow.
Lets consider the case,

1) Control plane programs the hardware with Queue termination flow.
2) Software dataplane programmed to treat the packets from the specific

queue accordingly.

quoted

3) Remove the flow from the hardware. (Lets consider this is a long wait

process..).

quoted

Or even there is a chance that hardware take more time to report the
status than removing it physically . Now the packets in the queue no longer

consider as matched/flow hit.

quoted

. This is due to the software dataplane update is yet to happen.
We must need a way to sync between software datapath and classifier
APIs even though they are both programmed from a different control

thread.

quoted

Are we saying these APIs are only meant for user defined static flows??

No, that is definitely not the intent. These are good points.

With the specified API, applications may have to adapt their logic and take
extra precautions in order to remain on the safe side at all times.

For your above example, the application cannot assume a rule is
added/deleted as long as the PMD has not completed the related operation,
which means keeping the SW rule/fallback in place in the meantime. Should
handle security concerns as long as after removing a rule, packets end up in a
default queue entirely processed by SW. Obviously this may worsen
response time.

The ID action can help with this. By knowing which rule a received packet is
associated with, processing can be temporarily offloaded by another thread
without much complexity.

[Sugesh] Setting ID for every flow may not viable especially when the size of ID
is small(just only 8 bits). I am not sure is this a valid case though.

Agreed, I'm not saying this solution works for all devices, particularly
those that do not support ID at all.

How about a hardware flow flag in packet descriptor that set when the
packets hits any hardware rule. This way software doesn’t worry /blocked by a
hardware rule . Even though there is an additional overhead of validating this flag,
software datapath can identify the hardware processed packets easily.
This way the packets traverses the software fallback path until the rule configuration is
complete. This flag avoids setting ID action for every hardware flow that are configuring.

That makes sense. I see it as a sort of single bit ID but it could be
implemented through a different action for less capable devices. PMDs that
support 32 bit IDs could reuse the same code for both features.

I understand you'd prefer having this feature always present, however we
already know that not all PMDs/devices support it, and like everything else
this is a kind of offload that needs to be explicitly requested by the
application as it may not be needed.

If we go with the separate action, then perhaps it would make sense to
rename "ID" to "MARK" to make things clearer:

 RTE_FLOW_ACTION_TYPE_FLAG /* Flag packets processed by flow rule. */

 RTE_FLOW_ACTION_TYPE_MARK /* Attach a 32 bit value to a packet. */

I guess the result of the FLAG action would be something in ol_flag.

Thoughts?

quoted

I think applications have to implement SW fallbacks all the time, as even
some sort of guarantee on the flow rule processing time may not be enough
to avoid misdirected packets and related security issues.

[Sugesh] Software fallback will be there always. However I am little bit confused on
the way software going to identify the packets that are already hardware processed . I feel we need some
notification in the packet itself, when a hardware rule hits. ID/flag/any other options?

Yeah I think so too, as long as it is optional because we cannot assume all
PMDs will support it.

quoted

Let's wait for applications to start using this API and then consider an extra
set of asynchronous / real-time functions when the need arises. It should not
impact the way rules are specified

[Sugesh] Sure. I think the rule definition may not impact with this.

Thanks for your comments.

-- 
Adrien Mazarguil
6WIND