Re: [PATCH] IB/ipoib: move back the IB LL address into the hard header

On Tue, Oct 11, 2016 at 07:15:44PM +0200, Paolo Abeni wrote:

Also the connected mode maximum mtu is reduced by 16 bytes to
cope with the increased hard header len.

Changing the MTU is going to cause annoying interop problems, can you
avoid this?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, 2016-10-11 at 11:32 -0600, Jason Gunthorpe wrote:

On Tue, Oct 11, 2016 at 07:15:44PM +0200, Paolo Abeni wrote:

quoted

Also the connected mode maximum mtu is reduced by 16 bytes to
cope with the increased hard header len.

Changing the MTU is going to cause annoying interop problems, can you
avoid this?

I don't like changing the maximum MTU value, too, but I was unable to
find an alternative solution. The PMTU detection should protect against
such issues. 

On 10/11/2016 1:32 PM, Jason Gunthorpe wrote:

On Tue, Oct 11, 2016 at 07:15:44PM +0200, Paolo Abeni wrote:

quoted

Also the connected mode maximum mtu is reduced by 16 bytes to
cope with the increased hard header len.

Changing the MTU is going to cause annoying interop problems, can you
avoid this?

(Paolo did the work I'm describing here, I'm just giving the explanation
he gave me):

Not using this particular solution I don't think.  We tried it without
increasing the declared hard header length and it broke when dealing
with skb_clone/GSO paths.  In order to make the LL pseudo header get
copied along with the rest of the encap and data on clone, we had to
declare the header.  The problem then became that the sg setup is such
that we are limited to 16 4k pages for the sg array, so that header had
to come out of the 64k maximum mtu.

-- 
Doug Ledford [off-list ref]
    GPG Key ID: 0E572FDD

On Tue, Oct 11, 2016 at 01:41:56PM -0400, Doug Ledford wrote:

declare the header.  The problem then became that the sg setup is such
that we are limited to 16 4k pages for the sg array, so that header had
to come out of the 64k maximum mtu.

Oh, that clarifies things..

Hum, so various options become:
 - Use >=17 SGL entries when creating the QP. Is this possible
   on common adapters?
 - Use the FRWR infrastructure when necessary. Is there any chance
   the majority of skbs will have at least two physically
   continuous pages to make this overhead rare? Perhaps as a fall
   back if many adaptors can do >=17 SGLs 
 - Pad the hard header out to 4k and discard the first page
   when building the sgl
 - Memcopy the first ~8k into a contiguous 8k region on send
 - Move the pseudo header to the end so it can cross the page
   barrier without needing a sgl entry. (probably impossible?)
 

From Paolo

AFAICS the max mtu is already underlying h/w dependent, how does such
differences are currently coped by ? (I'm sorry I lack some/a lot of IB
back-ground)

It isn't h/w dependent. In CM mode the MTU is 65520 because that is
what is hard coded into the ipoib driver. We tell everyone to use that
number. Eg see RH's docs on the subject:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Configuring_IPoIB.html

AFAIK, today everyone just wires that number into their scripts, so we
have to mass change everything to the smaller number. That sounds
really hard, IMHO if there is any way to avoid it we should, even if
it is a little costly.

Jason

On Tue, 2016-10-11 at 12:01 -0600, Jason Gunthorpe wrote:

quoted

AFAICS the max mtu is already underlying h/w dependent, how does such
differences are currently coped by ? (I'm sorry I lack some/a lot of IB
back-ground)

It isn't h/w dependent. In CM mode the MTU is 65520 because that is
what is hard coded into the ipoib driver. We tell everyone to use that
number. Eg see RH's docs on the subject:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Configuring_IPoIB.html

AFAIK, today everyone just wires that number into their scripts, so we
have to mass change everything to the smaller number. That sounds
really hard, IMHO if there is any way to avoid it we should, even if
it is a little costly.

Thank you for the details!

The first s/g fragment (the head buffer) is not allocated with the page
allocator, so perhaps there is some not too difficult/costly way out of
this.