[PATCH V3 net-next 1/1] hv_netvsc: Use the xmit_more skb flag to optimize signaling the host

Based on the information given to this driver (via the xmit_more skb flag),
we can defer signaling the host if more packets are on the way. This will help
make the host more efficient since it can potentially process a larger batch of
packets. Implement this optimization.

Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
	v2: Fixed up indentation based on feedback from David Miller.
	v3: If the queue is stopped, deal with that condition: Eric Dumazet [off-list ref]
 drivers/net/hyperv/netvsc.c     |   20 ++++++++++++--------
 drivers/net/hyperv/netvsc_drv.c |    2 ++
 2 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/drivers/net/hyperv/netvsc.c b/drivers/net/hyperv/netvsc.c
index ea091bc..36fef17 100644
--- a/drivers/net/hyperv/netvsc.c
+++ b/drivers/net/hyperv/netvsc.c

@@ -743,6 +743,7 @@ static inline int netvsc_send_pkt(
 	u64 req_id;
 	int ret;
 	struct hv_page_buffer *pgbuf;
+	u32 vmbus_flags = VMBUS_DATA_PACKET_FLAG_COMPLETION_REQUESTED;
 
 	nvmsg.hdr.msg_type = NVSP_MSG1_TYPE_SEND_RNDIS_PKT;
 	if (packet->is_data_pkt) {

@@ -772,19 +773,22 @@ static inline int netvsc_send_pkt(
 	if (packet->page_buf_cnt) {
 		pgbuf = packet->cp_partial ? packet->page_buf +
 			packet->rmsg_pgcnt : packet->page_buf;
-		ret = vmbus_sendpacket_pagebuffer(out_channel,
-						  pgbuf,
-						  packet->page_buf_cnt,
-						  &nvmsg,
-						  sizeof(struct nvsp_message),
-						  req_id);
+		ret = vmbus_sendpacket_pagebuffer_ctl(out_channel,
+						      pgbuf,
+						      packet->page_buf_cnt,
+						      &nvmsg,
+						      sizeof(struct
+							     nvsp_message),
+						      req_id,
+						      vmbus_flags,
+						      !packet->xmit_more);
 	} else {
-		ret = vmbus_sendpacket(
+		ret = vmbus_sendpacket_ctl(
 				out_channel, &nvmsg,
 				sizeof(struct nvsp_message),
 				req_id,
 				VM_PKT_DATA_INBAND,
-				VMBUS_DATA_PACKET_FLAG_COMPLETION_REQUESTED);
+				vmbus_flags, !packet->xmit_more);
 	}
 
 	if (ret == 0) {

diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index 5993c7e..4efaa6e 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c

@@ -435,6 +435,8 @@ check_size:
 	packet->page_buf = page_buf;
 
 	packet->q_idx = skb_get_queue_mapping(skb);
+	if (netif_tx_queue_stopped(netdev_get_tx_queue(net, packet->q_idx)))
+		packet->xmit_more = false;
 
 	packet->is_data_pkt = true;
 	packet->total_data_buflen = skb->len;

-- 
1.7.4.1

On Wed, 2015-05-06 at 11:20 -0700, K. Y. Srinivasan wrote:

Based on the information given to this driver (via the xmit_more skb flag),
we can defer signaling the host if more packets are on the way. This will help
make the host more efficient since it can potentially process a larger batch of
packets. Implement this optimization.

Have you tested this condition ever triggers ?

It should not.

netvsc_start_xmit() should not be called if the queue is stopped.

The problem is the following :

netvsc_start_xmit() is called with the packet that is going to stop the
queue, filling last slot, but with skb->xmit_more = 1;

So you need to do something about xmit_more, after calling
netif_tx_stop_queue(), not before.

-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
Sent: Wednesday, May 6, 2015 11:09 AM
To: KY Srinivasan
Cc: davem@davemloft.net; netdev@vger.kernel.org; linux-
kernel@vger.kernel.org; devel@linuxdriverproject.org; olaf@aepfle.de;
apw@canonical.com; jasowang@redhat.com
Subject: Re: [PATCH V3 net-next 1/1] hv_netvsc: Use the xmit_more skb flag
to optimize signaling the host

On Wed, 2015-05-06 at 11:20 -0700, K. Y. Srinivasan wrote:

quoted

Based on the information given to this driver (via the xmit_more skb flag),
we can defer signaling the host if more packets are on the way. This will

help

quoted

make the host more efficient since it can potentially process a larger batch

of

quoted

packets. Implement this optimization.

quoted

diff --git a/drivers/net/hyperv/netvsc_drv.c

b/drivers/net/hyperv/netvsc_drv.c

quoted

index 5993c7e..4efaa6e 100644

--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c

@@ -435,6 +435,8 @@ check_size:
 	packet->page_buf = page_buf;

 	packet->q_idx = skb_get_queue_mapping(skb);
+	if (netif_tx_queue_stopped(netdev_get_tx_queue(net, packet-

q_idx)))
+		packet->xmit_more = false;

Have you tested this condition ever triggers ?

It should not.

netvsc_start_xmit() should not be called if the queue is stopped.

The problem is the following :

netvsc_start_xmit() is called with the packet that is going to stop the
queue, filling last slot, but with skb->xmit_more = 1;

So you need to do something about xmit_more, after calling
netif_tx_stop_queue(), not before.

Ah! I too was wondering how we could get into this situation. The condition you mention
is already handled in the lower level - if the attempt to put the last packet on vmbus were to
fail because the ring is full, we will notify the host independent of kick_q parameter - look
at the function vmbus_sendpacket_ctl() in drivers/hv/channel.c

I will resend the patch by reverting this to version 2.

Regards,

K. Y

On Wed, 2015-05-06 at 18:28 +0000, KY Srinivasan wrote:

Ah! I too was wondering how we could get into this situation. The condition you mention
is already handled in the lower level - if the attempt to put the last packet on vmbus were to
fail because the ring is full, we will notify the host independent of kick_q parameter - look
at the function vmbus_sendpacket_ctl() in drivers/hv/channel.c

I will resend the patch by reverting this to version 2.

I am not really convinced by what you are saying ;)

problem is not when you fail to add an additional packet.

Problem is :

You queue a packet but not kick because skb->xmit_more = 1, and _then_
you call netif_tx_stop_queue().

No further packet will be delivered to your driver, until
netif_tx_wake_queue() is called again from netvsc_send_completion(),
possibly milli seconds later.

If qdisc was replaced by another one, no packet will kick again, unless
other traffic comes.

Given complexity of your driver, I have no easy way to check you made
this xmit_more conversion correctly, and one packet can not sit in the
equivalent of TX ring buffer for unbound time.

Better add a full explanation into your changelog.

-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
Sent: Wednesday, May 6, 2015 12:18 PM
To: KY Srinivasan
Cc: davem@davemloft.net; netdev@vger.kernel.org; linux-
kernel@vger.kernel.org; devel@linuxdriverproject.org; olaf@aepfle.de;
apw@canonical.com; jasowang@redhat.com
Subject: Re: [PATCH V3 net-next 1/1] hv_netvsc: Use the xmit_more skb flag
to optimize signaling the host

On Wed, 2015-05-06 at 18:28 +0000, KY Srinivasan wrote:

quoted

Ah! I too was wondering how we could get into this situation. The condition

you mention

quoted

is already handled in the lower level - if the attempt to put the last packet

on vmbus were to

quoted

fail because the ring is full, we will notify the host independent of kick_q

parameter - look

quoted

at the function vmbus_sendpacket_ctl() in drivers/hv/channel.c

I will resend the patch by reverting this to version 2.

I am not really convinced by what you are saying ;)

problem is not when you fail to add an additional packet.

Problem is :

You queue a packet but not kick because skb->xmit_more = 1, and _then_
you call netif_tx_stop_queue().

No further packet will be delivered to your driver, until
netif_tx_wake_queue() is called again from netvsc_send_completion(),
possibly milli seconds later.

Ok; I see your point. We stop the q in our driver under two conditions:

1. When the ring is full and we fail to queue. This situation is taken care of already.
2. We fall below a certain free space in the ring buffer - this is the case you are pointing out.

I will address this and resend the patch. Thanks for the review.

Regards,

K. Y

If qdisc was replaced by another one, no packet will kick again, unless
other traffic comes.

Given complexity of your driver, I have no easy way to check you made
this xmit_more conversion correctly, and one packet can not sit in the
equivalent of TX ring buffer for unbound time.

Better add a full explanation into your changelog.