Re: i40e xmit path HW limitation

From: Vladislav Zolotarov <hidden>
Date: 2015-07-30 19:25:06

On Jul 30, 2015 22:00, "Zhang, Helin" [off-list ref] wrote:

quoted

-----Original Message-----
From: Vlad Zolotarov [mailto:vladz@cloudius-systems.com]
Sent: Thursday, July 30, 2015 10:56 AM
To: Zhang, Helin; Ananyev, Konstantin
Cc: dev@dpdk.org
Subject: Re: i40e xmit path HW limitation



On 07/30/15 20:33, Zhang, Helin wrote:

quoted

-----Original Message-----
From: Vlad Zolotarov [mailto:vladz@cloudius-systems.com]
Sent: Thursday, July 30, 2015 9:44 AM
To: Zhang, Helin; Ananyev, Konstantin
Cc: dev@dpdk.org
Subject: Re: i40e xmit path HW limitation



On 07/30/15 19:10, Zhang, Helin wrote:

quoted

-----Original Message-----
From: Vlad Zolotarov [mailto:vladz@cloudius-systems.com]
Sent: Thursday, July 30, 2015 7:58 AM
To: dev@dpdk.org; Ananyev, Konstantin; Zhang, Helin
Subject: RFC: i40e xmit path HW limitation

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver)
which is not handled in any way by a DPDK driver.
   From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data
descriptors per packet including both the header and payload

buffers).

quoted

• The total number of data descriptors for the whole TSO (explained
later on in this chapter) is unlimited as long as each segment
within the TSO obeys the previous rule (up to 8 data descriptors
per segment for both the TSO header and the segment payload

buffers)."

quoted

Yes, I remember the RX side just supports 5 segments per packet

receiving.

quoted

But what's the possible issue you thought about?

Note that it's a Tx size we are talking about.

See 30520831f058cd9d75c0f6b360bc5c5ae49b5f27 commit in linux net-next

repo.

quoted

If such a cluster arrives and you post it on the HW ring - HW will
shut this HW ring down permanently. The application will see that

it's ring is

quoted

stuck.

quoted

That issue was because of using more than 8 descriptors for a packet

for TSO.

quoted

There is no problem in transmitting the TSO packet with more than 8

fragments.

quoted

On the opposite - one can't transmit a non-TSO packet with more than 8
fragments.
One also can't transmit the TSO packet that would contain more than 8

fragments

quoted

in a single TSO segment including the TSO headers.

Pls., read the HW spec as I quoted above for more details.

I meant a packet to be transmitted by the hardware, but not the TSO

packet in memory.

It could be a segment in TSO packet in memory.
The linearize check in kernel driver is not for TSO only, it is for both

TSO and

NON-TSO cases.

That's what i was trying to tell u. Great we are on the same page at
last... 😉

quoted

This means that, for instance, long cluster with small fragments
has to be linearized before it may be placed on the HW ring.

What type of size of the small fragments? Basically 2KB is the
default size of

mbuf of most

quoted

example applications. 2KB x 8 is bigger than 1.5KB. So it is enough
for the

maximum

quoted

packet size we supported.
If 1KB mbuf is used, don't expect it can transmit more than 8KB

size of

quoted

packet.

quoted

I kinda lost u here. Again, we talk about the Tx side here and
buffers are not obligatory completely filled. Namely there may be a
cluster with
15 fragments 100 bytes each.

The root cause is using more than 8 descriptors for a packet.

That would be if u would like to SUPER simplify the HW limitation above.
In that case u would significantly limit the different packets that may

be sent

quoted

without the linearization.

quoted

Linux driver can help
on reducing number of descriptors to be used by merging small size of
payload together, right?
It is not for TSO, it is just for packet transmitting. 2 options in

my mind:

quoted

1. Use should ensure it will not use more than 8 descriptors per

packet for

quoted

transmitting.

This requirement is too restricting. Pls., see above.

quoted

2. DPDK driver should try to merge small packet together for such

case, like

quoted

Linux kernel driver.

quoted

I prefer to use option 1, users should ensure that in the application
or up layer software, and keep the PMD driver as simple as possible.

The above statement is super confusing: on the one hand u suggest the

DPDK

quoted

driver to merge the small packet (fragments?) together (how?) and then u
immediately propose the user application to do that. Could u, pls.,

clarify what

quoted

exactly u suggest here?
If that's to leave it to the application - note that it would demand

patching all

quoted

existing DPDK applications that send TCP packets.

Those are two of obvious options. One is to do that in PMD, the other one

is to do

that in up layer. I did not mean it needs to do both!

Ok. I just didn't understand where the (2) description ends. Now i get u...
😉

quoted

But I have a thought that the maximum number of RX/TX descriptor
should be able to be queried somewhere.

There is no such thing as maximum number of Tx fragments in a TSO case.
It's only limited by the Tx ring size.

Again, it is not for TSO case only. You are talking about how to

implement it?

I understand that and what I was trying to tell was that any limit we
choose that satisfies the non-TSO case would be too restricting  for a TSO
case. Therefore I'd suggest to go to the second option and implement the
merging in the driver. Not only it would be the cleanest and robust way but
it would also prevent the tremendous code duplication across all
applications susceptible to this HW limitation.

Anything missed can be added, as long as it is reasonable.

Regards,
Helin

quoted

Regards,
Helin

quoted

In more standard environments like Linux or FreeBSD drivers the
solution is straight forward - call skb_linearize()/m_collapse()

corresponding.

quoted

In the non-conformist environment like DPDK life is not that easy -
there is no easy way to collapse the cluster into a linear buffer
from inside the device

driver

quoted

since device driver doesn't allocate memory in a fast path and
utilizes the user allocated pools only.
Here are two proposals for a solution:

   1. We may provide a callback that would return a user TRUE if a

give

quoted

      cluster has to be linearized and it should always be called

before

quoted

      rte_eth_tx_burst(). Alternatively it may be called from

inside the

quoted

      rte_eth_tx_burst() and rte_eth_tx_burst() is changed to

return

quoted

some

quoted

      error code for a case when one of the clusters it's given

has to be

quoted

      linearized.
   2. Another option is to allocate a mempool in the driver with

the

quoted

      elements consuming a single page each (standard 2KB buffers

would

quoted

      do). Number of elements in the pool should be as Tx ring

length

quoted

      multiplied by "64KB/(linear data length of the buffer in the

pool

quoted

      above)". Here I use 64KB as a maximum packet length and not

taking

quoted

      into an account esoteric things like "Giant" TSO mentioned

in the

quoted

      spec above. Then we may actually go and linearize the

cluster if

quoted

      needed on top of the buffers from the pool above, post the

buffer

quoted

      from the mempool above on the HW ring, link the original

cluster to

quoted

      that new cluster (using the private data) and release it

when the

quoted

      send is done.


The first is a change in the API and would require from the
application some additional handling (linearization). The second
would require some additional memory but would keep all dirty
details inside the driver and would leave the rest of the code

intact.

quoted
quoted
quoted
quoted
quoted
Pls., comment.

thanks,
vlad

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help