Re: i40e xmit path HW limitation

On 07/30/15 20:33, Zhang, Helin wrote:

quoted

-----Original Message-----
From: Vlad Zolotarov [mailto:vladz@cloudius-systems.com]
Sent: Thursday, July 30, 2015 9:44 AM
To: Zhang, Helin; Ananyev, Konstantin
Cc: dev@dpdk.org
Subject: Re: i40e xmit path HW limitation



On 07/30/15 19:10, Zhang, Helin wrote:

quoted

quoted

-----Original Message-----
From: Vlad Zolotarov [mailto:vladz@cloudius-systems.com]
Sent: Thursday, July 30, 2015 7:58 AM
To: dev@dpdk.org; Ananyev, Konstantin; Zhang, Helin
Subject: RFC: i40e xmit path HW limitation

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver)
which is not handled in any way by a DPDK driver.
   From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data
descriptors per packet including both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained
later on in this chapter) is unlimited as long as each segment within
the TSO obeys the previous rule (up to 8 data descriptors per segment
for both the TSO header and the segment payload buffers)."

Yes, I remember the RX side just supports 5 segments per packet receiving.
But what's the possible issue you thought about?

Note that it's a Tx size we are talking about.

See 30520831f058cd9d75c0f6b360bc5c5ae49b5f27 commit in linux net-next repo.
If such a cluster arrives and you post it on the HW ring - HW will shut this HW ring
down permanently. The application will see that it's ring is stuck.

That issue was because of using more than 8 descriptors for a packet for TSO.

There is no problem in transmitting the TSO packet with more than 8 fragments.
On the opposite - one can't transmit a non-TSO packet with more than 8 fragments.
One also can't transmit the TSO packet that would contain more than 8 fragments in a single TSO segment including the TSO headers.

Pls., read the HW spec as I quoted above for more details.

quoted

quoted

quoted

This means that, for instance, long cluster with small fragments has to be
linearized before it may be placed on the HW ring.

What type of size of the small fragments? Basically 2KB is the default size of

mbuf of most

quoted

example applications. 2KB x 8 is bigger than 1.5KB. So it is enough for the

maximum

quoted

packet size we supported.
If 1KB mbuf is used, don't expect it can transmit more than 8KB size of packet.

I kinda lost u here. Again, we talk about the Tx side here and buffers
are not obligatory completely filled. Namely there may be a cluster with
15 fragments 100 bytes each.

The root cause is using more than 8 descriptors for a packet.

That would be if u would like to SUPER simplify the HW limitation above. In that case u would significantly limit the different packets that may be sent without the linearization.

Linux driver can help
on reducing number of descriptors to be used by merging small size of payload
together, right?
It is not for TSO, it is just for packet transmitting. 2 options in my mind:
1. Use should ensure it will not use more than 8 descriptors per packet for transmitting.

This requirement is too restricting. Pls., see above.

2. DPDK driver should try to merge small packet together for such case, like Linux kernel driver.
I prefer to use option 1, users should ensure that in the application or up layer software,
and keep the PMD driver as simple as possible.

The above statement is super confusing: on the one hand u suggest the DPDK driver to merge the small packet (fragments?) together (how?) and then u immediately propose the user application to do that. Could u, pls., clarify what exactly u suggest here?
If that's to leave it to the application - note that it would demand patching all existing DPDK applications that send TCP packets.

But I have a thought that the maximum number of RX/TX descriptor should be able to be
queried somewhere.

There is no such thing as maximum number of Tx fragments in a TSO case. It's only limited by the Tx ring size.

Regards,
Helin

quoted

quoted

quoted

In more standard environments like Linux or FreeBSD drivers the solution is
straight forward - call skb_linearize()/m_collapse() corresponding.
In the non-conformist environment like DPDK life is not that easy - there is no
easy way to collapse the cluster into a linear buffer from inside the device

driver

quoted

quoted

since device driver doesn't allocate memory in a fast path and utilizes the user
allocated pools only.
Here are two proposals for a solution:

   1. We may provide a callback that would return a user TRUE if a give
      cluster has to be linearized and it should always be called before
      rte_eth_tx_burst(). Alternatively it may be called from inside the
      rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return some
      error code for a case when one of the clusters it's given has to be
      linearized.
   2. Another option is to allocate a mempool in the driver with the
      elements consuming a single page each (standard 2KB buffers would
      do). Number of elements in the pool should be as Tx ring length
      multiplied by "64KB/(linear data length of the buffer in the pool
      above)". Here I use 64KB as a maximum packet length and not taking
      into an account esoteric things like "Giant" TSO mentioned in the
      spec above. Then we may actually go and linearize the cluster if
      needed on top of the buffers from the pool above, post the buffer
      from the mempool above on the HW ring, link the original cluster to
      that new cluster (using the private data) and release it when the
      send is done.


The first is a change in the API and would require from the application some
additional handling (linearization). The second would require some additional
memory but would keep all dirty details inside the driver and would leave the
rest of the code intact.

Pls., comment.

thanks,
vlad