-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
Sent: Tuesday, November 3, 2015 11:37 AM
To: David Miller <davem@davemloft.net>
Cc: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
[off-list ref]; edumazet@google.com;
netdev@vger.kernel.org
Subject: Re: [PATCH net-next] net: increase LL_MAX_HEADER if HYPERV_NET
is enabled

On Tue, 2015-11-03 at 10:33 -0500, David Miller wrote:

quoted

From: KY Srinivasan <kys@microsoft.com>
Date: Tue, 3 Nov 2015 07:59:36 +0000

quoted

I have implemented the scheme we had discussed a few weeks ago. In
this new implementation our driver is NOT requesting addition
headroom - rndis header and the per packet state is being maintained
outside of the skb. What I am seeing is that when I have
LL_MAX_HEADER set to 220 bytes, even though our driver is not using
the additional head room, I see about a 10% boost in the peak
performance (about 34 Gbps on a 40Gbps interface). However, when I
set the LL_MAX_HEADER value to the current default, the peak
performance drops back to what we currently have (around 31 Gbps).
In both these cases, there is no reallocation of skb since no
additional headroom is being requested and yet there is a
significant difference in performance.  I trying to figure out why
this is the case, your insights will be greatly appreciated.

It probably has something to do with cache line or data alignment.

This also might be because of a slight change in skb->truesize, and/or a
change of amount of payload in skb->head

(Increasing LL_MAX_HEADER is reducing amount of payload in skb->head)

Can't you run perf tool to get some precise profiling ?


Another red flag in you driver xmit is :

return (ret == -EAGAIN) ? NETDEV_TX_BUSY : NETDEV_TX_OK;


extract from include/linux/netdevice.h
 * netdev_tx_t (*ndo_start_xmit)(struct sk_buff *skb,
 *                               struct net_device *dev);
 *      Called when a packet needs to be transmitted.
 *      Returns NETDEV_TX_OK.  Can return NETDEV_TX_BUSY, but you should
stop
 *      the queue before that can happen; it's for obsolete devices and weird
 *      corner cases, but the stack really does a non-trivial amount
 *      of useless work if you return NETDEV_TX_BUSY.

We stop xmit when ring buffer is <10% available (netvsc_send_pkt()), so we 
almost never hit empty ring buffer and return NETDEV_TX_BUSY. 
But we still keep this busy return in our code, just for "weird corner cases".

Thanks,
- Haiyang