[PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

Subsystems: networking drivers, the rest, xen hypervisor interface

STALE3488d REVIEWED: 7 (7M)

1 review trailer (1 from subsystem maintainers).

11 messages, 4 authors, 2017-01-31 · open the first message on its own page

[PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: Remanan Pillai <hidden>
Date: 2017-01-18 20:25:49

From: Vineeth Remanan Pillai <redacted>

During an OOM scenario, request slots could not be created as skb
allocation fails. So the netback cannot pass in packets and netfront
wrongly assumes that there is no more work to be done and it disables
polling. This causes Rx to stall.

The issue is with the retry logic which schedules the timer if the
created slots are less than NET_RX_SLOTS_MIN. The count of new request
slots to be pushed are calculated as a difference between new req_prod
and rsp_cons which could be more than the actual slots, if there are
unconsumed responses.

The fix is to calculate the count of newly created slots as the
difference between new req_prod and old req_prod.

Signed-off-by: Vineeth Remanan Pillai <redacted>
Reviewed-by: Juergen Gross <jgross@suse.com>
---
Changes in v2:
	- Removed the old implementation of enabling polling on
	  skb allocation error.
	- Corrected the refill timer logic to schedule when newly
	  created slots since last push is less than NET_RX_SLOTS_MIN.

 drivers/net/xen-netfront.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index 40f26b6..2c7c29f 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -321,7 +321,7 @@ static void xennet_alloc_rx_buffers(struct netfront_queue *queue)
 	queue->rx.req_prod_pvt = req_prod;
 
 	/* Not enough requests? Try again later. */
-	if (req_prod - queue->rx.rsp_cons < NET_RX_SLOTS_MIN) {
+	if (req_prod - queue->rx.sring->req_prod < NET_RX_SLOTS_MIN) {
 		mod_timer(&queue->rx_refill_timer, jiffies + (HZ/10));
 		return;
 	}
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: Vineeth Remanan Pillai <hidden>
Date: 2017-01-19 16:38:06

From: Vineeth Remanan Pillai <redacted>

During an OOM scenario, request slots could not be created as skb
allocation fails. So the netback cannot pass in packets and netfront
wrongly assumes that there is no more work to be done and it disables
polling. This causes Rx to stall.

The issue is with the retry logic which schedules the timer if the
created slots are less than NET_RX_SLOTS_MIN. The count of new request
slots to be pushed are calculated as a difference between new req_prod
and rsp_cons which could be more than the actual slots, if there are
unconsumed responses.

The fix is to calculate the count of newly created slots as the
difference between new req_prod and old req_prod.

Signed-off-by: Vineeth Remanan Pillai <redacted>
Reviewed-by: Juergen Gross <jgross@suse.com>
---
Changes in v2:
	- Removed the old implementation of enabling polling on
	  skb allocation error.
	- Corrected the refill timer logic to schedule when newly
	  created slots since last push is less than NET_RX_SLOTS_MIN.

  drivers/net/xen-netfront.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index 40f26b6..2c7c29f 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -321,7 +321,7 @@ static void xennet_alloc_rx_buffers(struct netfront_queue *queue)
  	queue->rx.req_prod_pvt = req_prod;
  
  	/* Not enough requests? Try again later. */
-	if (req_prod - queue->rx.rsp_cons < NET_RX_SLOTS_MIN) {
+	if (req_prod - queue->rx.sring->req_prod < NET_RX_SLOTS_MIN) {
  		mod_timer(&queue->rx_refill_timer, jiffies + (HZ/10));
  		return;
  	}
-- 
2.7.4

Re: [PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: David Miller <davem@davemloft.net>
Date: 2017-01-19 17:11:40

From: Vineeth Remanan Pillai <redacted>
Date: Thu, 19 Jan 2017 08:35:39 -0800
From: Vineeth Remanan Pillai <redacted>

During an OOM scenario, request slots could not be created as skb
allocation fails. So the netback cannot pass in packets and netfront
wrongly assumes that there is no more work to be done and it disables
polling. This causes Rx to stall.

The issue is with the retry logic which schedules the timer if the
created slots are less than NET_RX_SLOTS_MIN. The count of new request
slots to be pushed are calculated as a difference between new req_prod
and rsp_cons which could be more than the actual slots, if there are
unconsumed responses.

The fix is to calculate the count of newly created slots as the
difference between new req_prod and old req_prod.

Signed-off-by: Vineeth Remanan Pillai <redacted>
Reviewed-by: Juergen Gross <jgross@suse.com>
---
Changes in v2:
	- Removed the old implementation of enabling polling on
	  skb allocation error.
	- Corrected the refill timer logic to schedule when newly
	  created slots since last push is less than NET_RX_SLOTS_MIN.
Your postings aren't showing up on vger.kernel.org at all.

Are you getting a bounce message back?  I can only assume you are triggering
one of the various content filters we have.

Re: [PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: David Miller <davem@davemloft.net>
Date: 2017-01-19 18:12:47

From: Vineeth Remanan Pillai <redacted>
Date: Thu, 19 Jan 2017 09:17:09 -0800
Should I try sending it once again?
No need, it just showed up.

Re: [PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: Vineeth Remanan Pillai <hidden>
Date: 2017-01-19 18:24:57


On 01/19/2017 09:11 AM, David Miller wrote:
From: Vineeth Remanan Pillai <redacted>
Date: Thu, 19 Jan 2017 08:35:39 -0800
quoted
From: Vineeth Remanan Pillai <redacted>

During an OOM scenario, request slots could not be created as skb
allocation fails. So the netback cannot pass in packets and netfront
wrongly assumes that there is no more work to be done and it disables
polling. This causes Rx to stall.

The issue is with the retry logic which schedules the timer if the
created slots are less than NET_RX_SLOTS_MIN. The count of new request
slots to be pushed are calculated as a difference between new req_prod
and rsp_cons which could be more than the actual slots, if there are
unconsumed responses.

The fix is to calculate the count of newly created slots as the
difference between new req_prod and old req_prod.

Signed-off-by: Vineeth Remanan Pillai <redacted>
Reviewed-by: Juergen Gross <jgross@suse.com>
---
Changes in v2:
	- Removed the old implementation of enabling polling on
	  skb allocation error.
	- Corrected the refill timer logic to schedule when newly
	  created slots since last push is less than NET_RX_SLOTS_MIN.
Your postings aren't showing up on vger.kernel.org at all.

Are you getting a bounce message back?  I can only assume you are triggering
one of the various content filters we have.
I haven't received any bounce messages till now. The mail showed up
in xen-devel after about 8 hours yesterday. Not sure what is happening
with vger.kernel.org. My initial patch showed up in all the mailing 
list. The
only difference is, I switched to a machine running a later version of git.

Should I try sending it once again?

Thanks

Re: [PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: David Miller <davem@davemloft.net>
Date: 2017-01-20 19:09:16

From: Vineeth Remanan Pillai <redacted>
Date: Thu, 19 Jan 2017 08:35:39 -0800
From: Vineeth Remanan Pillai <redacted>

During an OOM scenario, request slots could not be created as skb
allocation fails. So the netback cannot pass in packets and netfront
wrongly assumes that there is no more work to be done and it disables
polling. This causes Rx to stall.

The issue is with the retry logic which schedules the timer if the
created slots are less than NET_RX_SLOTS_MIN. The count of new request
slots to be pushed are calculated as a difference between new req_prod
and rsp_cons which could be more than the actual slots, if there are
unconsumed responses.

The fix is to calculate the count of newly created slots as the
difference between new req_prod and old req_prod.

Signed-off-by: Vineeth Remanan Pillai <redacted>
Reviewed-by: Juergen Gross <jgross@suse.com>
---
Changes in v2:
	- Removed the old implementation of enabling polling on
	  skb allocation error.
	- Corrected the refill timer logic to schedule when newly
	  created slots since last push is less than NET_RX_SLOTS_MIN.
Applied.

Re: [PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Date: 2017-01-29 23:11:07


On 01/19/2017 11:35 AM, Vineeth Remanan Pillai wrote:
From: Vineeth Remanan Pillai <redacted>

During an OOM scenario, request slots could not be created as skb
allocation fails. So the netback cannot pass in packets and netfront
wrongly assumes that there is no more work to be done and it disables
polling. This causes Rx to stall.

The issue is with the retry logic which schedules the timer if the
created slots are less than NET_RX_SLOTS_MIN. The count of new request
slots to be pushed are calculated as a difference between new req_prod
and rsp_cons which could be more than the actual slots, if there are
unconsumed responses.

The fix is to calculate the count of newly created slots as the
difference between new req_prod and old req_prod.

Signed-off-by: Vineeth Remanan Pillai <redacted>
Reviewed-by: Juergen Gross <jgross@suse.com>
---
Changes in v2:
    - Removed the old implementation of enabling polling on
      skb allocation error.
    - Corrected the refill timer logic to schedule when newly
      created slots since last push is less than NET_RX_SLOTS_MIN.

There are couple of problems with this patch.
1. The 'if' clause now evaluates to true on pretty much every call to 
xennet_alloc_rx_buffers().
2. It tickles a latent bug during resume where the timer triggers before 
we re-connect. The trouble is that we now try to dereference 
queue->rx.sring which is NULL since we disconnect in netfront_resume(). 
(Curiously, I only observe it with 32-bit guests)

I'll send a patch later that will delete the timer since it looks like a 
bug to me in any case but the first problem seems to be more serious 
than the problem that this patch addresses.

-boris
quoted hunk
 drivers/net/xen-netfront.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index 40f26b6..2c7c29f 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -321,7 +321,7 @@ static void xennet_alloc_rx_buffers(struct
netfront_queue *queue)
     queue->rx.req_prod_pvt = req_prod;

     /* Not enough requests? Try again later. */
-    if (req_prod - queue->rx.rsp_cons < NET_RX_SLOTS_MIN) {
+    if (req_prod - queue->rx.sring->req_prod < NET_RX_SLOTS_MIN) {
         mod_timer(&queue->rx_refill_timer, jiffies + (HZ/10));
         return;
     }
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: Vineeth Remanan Pillai <hidden>
Date: 2017-01-30 16:48:54

On 01/29/2017 03:09 PM, Boris Ostrovsky wrote:
There are couple of problems with this patch.
1. The 'if' clause now evaluates to true on pretty much every call to 
xennet_alloc_rx_buffers().
Thanks for catching this. In my testing I did not notice this - mostly 
because of the nature of the workload in my testing.
2. It tickles a latent bug during resume where the timer triggers 
before we re-connect. The trouble is that we now try to dereference 
queue->rx.sring which is NULL since we disconnect in 
netfront_resume(). (Curiously, I only observe it with 32-bit guests)
I think we may hit this bug after removing the timer as well. We call 
RING_PUSH_REQUESTS_AND_CHECK_NOTIFY soon after, which also dereference 
queue->rx.sring.

Thanks,
Vineeth


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Date: 2017-01-30 17:07:25

On 01/30/2017 11:47 AM, Vineeth Remanan Pillai wrote:
quoted
2. It tickles a latent bug during resume where the timer triggers
before we re-connect. The trouble is that we now try to dereference
queue->rx.sring which is NULL since we disconnect in
netfront_resume(). (Curiously, I only observe it with 32-bit guests)
I think we may hit this bug after removing the timer as well. We call
RING_PUSH_REQUESTS_AND_CHECK_NOTIFY soon after, which also dereference
queue->rx.sring.

If the timer is deleted in xennet_disconnect_backend() then why would
anyone be pushing anything to the backend after that?

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: Vineeth Remanan Pillai <hidden>
Date: 2017-01-30 17:14:05

On 01/30/2017 09:06 AM, Boris Ostrovsky wrote:
On 01/30/2017 11:47 AM, Vineeth Remanan Pillai wrote:
quoted
quoted
2. It tickles a latent bug during resume where the timer triggers
before we re-connect. The trouble is that we now try to dereference
queue->rx.sring which is NULL since we disconnect in
netfront_resume(). (Curiously, I only observe it with 32-bit guests)
I think we may hit this bug after removing the timer as well. We call
RING_PUSH_REQUESTS_AND_CHECK_NOTIFY soon after, which also dereference
queue->rx.sring.
If the timer is deleted in xennet_disconnect_backend() then why would
anyone be pushing anything to the backend after that?
Sorry, I got the ordering wrong. Thanks for the clarification..

Thanks,
Vineeth

Re: [PATCH v2] xen-netfront: Fix Rx stall during network stress and OOM

From: Vineeth Remanan Pillai <hidden>
Date: 2017-01-31 16:53:39

On 01/30/2017 08:47 AM, Vineeth Remanan Pillai wrote:
On 01/29/2017 03:09 PM, Boris Ostrovsky wrote:
quoted
There are couple of problems with this patch.
1. The 'if' clause now evaluates to true on pretty much every call to 
xennet_alloc_rx_buffers().
Thanks for catching this. In my testing I did not notice this - mostly 
because of the nature of the workload in my testing.
I am working on a patch to revert to the old behavior and solve the Rx 
stall issue by scheduling the timer if any of the following
conditions are true:
  - unconsumed requests + new requests < NET_RX_SLOTS_MIN (old behavior)
  - skb allocations fail

Will send out the patch by next week after I can do some testing.

Thanks,
Vineeth


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help