We've seen this in Fedora for a long time now. As far as I know, nobody
really knows what is going on.
https://bugzilla.redhat.com/show_bug.cgi?id=717211
(Also, you probably should have sent this to netdev.)
CC netdev, Jay Cliburn [off-list ref],
It seems this driver has partial support for two TX rings.
(TX completion only drains the first ring)
Please try following patch.
We've seen this in Fedora for a long time now. As far as I know, nobody
really knows what is going on.
https://bugzilla.redhat.com/show_bug.cgi?id=717211
(Also, you probably should have sent this to netdev.)
CC netdev, Jay Cliburn [off-list ref],
It seems this driver has partial support for two TX rings.
(TX completion only drains the first ring)
Please try following patch.
We've seen this in Fedora for a long time now. As far as I know, nobody
really knows what is going on.
https://bugzilla.redhat.com/show_bug.cgi?id=717211
(Also, you probably should have sent this to netdev.)
CC netdev, Jay Cliburn [off-list ref],
It seems this driver has partial support for two TX rings.
(TX completion only drains the first ring)
Please try following patch.
@@ -2244,10 +2244,6 @@ static netdev_tx_t atl1c_xmit_frame(struct sk_buff *skb,dev_info(&adapter->pdev->dev,"tx locked\n");returnNETDEV_TX_LOCKED;}-if(skb->mark==0x01)-type=atl1c_trans_high;-else-type=atl1c_trans_normal;if(atl1c_tpd_avail(adapter,type)<tpd_req){/* no enough descriptor, just stop queue */
Hello Eric,
Thomas, have you had the chance to test this patch ?
Yes, I'm running 3.2.6 with your patch applied for 3 days now. I didn't see above warning yet, but the warning was rather seldom, so I'll keep an eye on it in the next weeks.
Thanks for the patch!
Kind regards
Thomas
From: Eric Dumazet <hidden> Date: 2012-02-16 06:18:11
Le jeudi 16 février 2012 à 07:13 +0100, Thomas Meyer a écrit :
Hello Eric,
quoted
Thomas, have you had the chance to test this patch ?
Yes, I'm running 3.2.6 with your patch applied for 3 days now. I didn't see above warning yet, but the warning was rather seldom, so I'll keep an eye on it in the next weeks.
Thanks for the patch!
Thanks for testing !
Do have any idea of what could set skb mark to 1 on some packets on your
setup ?
Some firewall rules or tc rules ?
From: Thomas Meyer <hidden> Date: 2012-02-16 06:24:32
Am 16.02.2012 um 07:18 schrieb Eric Dumazet [off-list ref]:
Le jeudi 16 février 2012 à 07:13 +0100, Thomas Meyer a écrit :
quoted
Hello Eric,
quoted
Thomas, have you had the chance to test this patch ?
Yes, I'm running 3.2.6 with your patch applied for 3 days now. I didn't see above warning yet, but the warning was rather seldom, so I'll keep an eye on it in the next weeks.
Thanks for the patch!
Thanks for testing !
Do have any idea of what could set skb mark to 1 on some packets on your
setup ?
No idea.
Some firewall rules or tc rules ?
I'm using a fedora 16 installation. No big changes made on the configuration.
The warning seems to occur only when shutting down the remote computer.
From: Eric Dumazet <hidden> Date: 2012-02-16 06:43:17
This driver attempts to use two TX rings but lacks proper support :
1) IRQ handler only takes care of TX completion on first TX ring
2) the stop/start logic uses the legacy functions (for non multiqueue
drivers)
This means all packets witk skb mark set to 1 are sent through high
queue but are never cleaned and queue eventualy fills and block the
device, triggering the infamous "NETDEV WATCHDOG" message.
Lets use a single TX ring to fix the problem, this driver is not a real
multiqueue one yet.
Minimal fix for stable kernels.
Reported-by: Thomas Meyer <redacted>
Tested-by: Thomas Meyer <redacted>
Signed-off-by: Eric Dumazet <redacted>
Cc: Jay Cliburn <redacted>
Cc: Chris Snook <chris.snook@gmail.com>
---
drivers/net/ethernet/atheros/atl1c/atl1c_main.c | 4 ----
1 file changed, 4 deletions(-)
Sorry, the piece of code that related to mark might be introduced by our internal test for multiple queues.
I remember mark could be changed by ip_table rule.
Thanks
Xiong
-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
On Behalf Of Eric Dumazet
Sent: Thursday, February 16, 2012 14:18
To: Thomas Meyer
Cc: Linux Kernel Mailing List; jcliburn@gmail.com; chris.snook@gmail.com;
netdev; Josh Boyer
Subject: Re: NETDEV WATCHDOG: eth0 (atl1c): transmit queue 0 timed out
Le jeudi 16 février 2012 à 07:13 +0100, Thomas Meyer a écrit :
quoted
Hello Eric,
quoted
Thomas, have you had the chance to test this patch ?
Yes, I'm running 3.2.6 with your patch applied for 3 days now. I didn't see
above warning yet, but the warning was rather seldom, so I'll keep an eye on it
in the next weeks.
quoted
Thanks for the patch!
Thanks for testing !
Do have any idea of what could set skb mark to 1 on some packets on your
setup ?
Some firewall rules or tc rules ?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a
message to majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html
On Thu, Feb 16, 2012 at 1:43 AM, Eric Dumazet [off-list ref] wrote:
This driver attempts to use two TX rings but lacks proper support :
1) IRQ handler only takes care of TX completion on first TX ring
2) the stop/start logic uses the legacy functions (for non multiqueue
drivers)
This means all packets witk skb mark set to 1 are sent through high
queue but are never cleaned and queue eventualy fills and block the
device, triggering the infamous "NETDEV WATCHDOG" message.
Lets use a single TX ring to fix the problem, this driver is not a real
multiqueue one yet.
Minimal fix for stable kernels.
Reported-by: Thomas Meyer <redacted>
Tested-by: Thomas Meyer <redacted>
Signed-off-by: Eric Dumazet <redacted>
Cc: Jay Cliburn <redacted>
Cc: Chris Snook <chris.snook@gmail.com>
As I think David handles netdev patches a bit differently for stable releases,
I'd like to suggest this get included in the next batch for the 3.2 kernel.
We've been seeing the bug this patch fixes in Fedora for quite a while now.
josh
From: Eric Dumazet <hidden> Date: 2012-02-16 13:12:40
Le jeudi 16 février 2012 à 07:36 -0500, Josh Boyer a écrit :
As I think David handles netdev patches a bit differently for stable releases,
I'd like to suggest this get included in the next batch for the 3.2 kernel.
We've been seeing the bug this patch fixes in Fedora for quite a while now.
From: David Miller <davem@davemloft.net> Date: 2012-02-20 00:00:22
From: Eric Dumazet <redacted>
Date: Thu, 16 Feb 2012 07:43:11 +0100
This driver attempts to use two TX rings but lacks proper support :
1) IRQ handler only takes care of TX completion on first TX ring
2) the stop/start logic uses the legacy functions (for non multiqueue
drivers)
This means all packets witk skb mark set to 1 are sent through high
queue but are never cleaned and queue eventualy fills and block the
device, triggering the infamous "NETDEV WATCHDOG" message.
Lets use a single TX ring to fix the problem, this driver is not a real
multiqueue one yet.
Minimal fix for stable kernels.
Reported-by: Thomas Meyer <redacted>
Tested-by: Thomas Meyer <redacted>
Signed-off-by: Eric Dumazet <redacted>
We've seen this in Fedora for a long time now. As far as I know, nobody
really knows what is going on.
https://bugzilla.redhat.com/show_bug.cgi?id=717211
(Also, you probably should have sent this to netdev.)
CC netdev, Jay Cliburn [off-list ref],
It seems this driver has partial support for two TX rings.
(TX completion only drains the first ring)
Please try following patch.
Thanks
This driver xmit function is racy I suspect, and several patches will be
needed to fix bugs.
For example, it uses a tx_lock, but no other part of the driver uses it.
It's a copy/paste leftover from a LLTX driver.
[PATCH] atl1c: remove useless tx lock
This lock has no purpose, since caller already runs in a serialized
context (its not a LLTX driver)
Signed-off-by: Eric Dumazet <redacted>
---
drivers/net/ethernet/atheros/atl1c/atl1c.h | 1 -
drivers/net/ethernet/atheros/atl1c/atl1c_main.c | 10 ----------
2 files changed, 11 deletions(-)
@@ -2239,16 +2237,10 @@ static netdev_tx_t atl1c_xmit_frame(struct sk_buff *skb,}tpd_req=atl1c_cal_tpd_req(skb);-if(!spin_trylock_irqsave(&adapter->tx_lock,flags)){-if(netif_msg_pktdata(adapter))-dev_info(&adapter->pdev->dev,"tx locked\n");-returnNETDEV_TX_LOCKED;-}if(atl1c_tpd_avail(adapter,type)<tpd_req){/* no enough descriptor, just stop queue */netif_stop_queue(netdev);-spin_unlock_irqrestore(&adapter->tx_lock,flags);returnNETDEV_TX_BUSY;}
@@ -2256,7 +2248,6 @@ static netdev_tx_t atl1c_xmit_frame(struct sk_buff *skb,/* do TSO and check sum */if(atl1c_tso_csum(adapter,skb,&tpd,type)!=0){-spin_unlock_irqrestore(&adapter->tx_lock,flags);dev_kfree_skb_any(skb);returnNETDEV_TX_OK;}
@@ -2239,24 +2239,20 @@ static netdev_tx_t atl1c_xmit_frame(struct sk_buff *skb,}tpd_req=atl1c_cal_tpd_req(skb);-if(!spin_trylock_irqsave(&adapter->tx_lock,flags)){-if(netif_msg_pktdata(adapter))-dev_info(&adapter->pdev->dev,"tx locked\n");-returnNETDEV_TX_LOCKED;-}if(atl1c_tpd_avail(adapter,type)<tpd_req){/* no enough descriptor, just stop queue */netif_stop_queue(netdev);-spin_unlock_irqrestore(&adapter->tx_lock,flags);-returnNETDEV_TX_BUSY;+smp_mb();+if(atl1c_tpd_avail(adapter,type)<tpd_req)+returnNETDEV_TX_BUSY;+netif_wake_queue(netdev);}tpd=atl1c_get_tpd(adapter,type);/* do TSO and check sum */if(atl1c_tso_csum(adapter,skb,&tpd,type)!=0){-spin_unlock_irqrestore(&adapter->tx_lock,flags);dev_kfree_skb_any(skb);returnNETDEV_TX_OK;}
Thanks
This driver xmit function is racy I suspect, and several patches will be
needed to fix bugs.
Here is a cumulative patch to hopefuly remove the races in this driver,
could you please test it ?
Hi,
just building a 3.2.7 kernel with your patch applied. I will watch out
for the warning in the next days.
many thanks for the patch!
kind regards
thomas
@@ -2239,24 +2239,20 @@ static netdev_tx_t atl1c_xmit_frame(struct sk_buff *skb,}tpd_req=atl1c_cal_tpd_req(skb);-if(!spin_trylock_irqsave(&adapter->tx_lock,flags)){-if(netif_msg_pktdata(adapter))-dev_info(&adapter->pdev->dev,"tx locked\n");-returnNETDEV_TX_LOCKED;-}if(atl1c_tpd_avail(adapter,type)<tpd_req){/* no enough descriptor, just stop queue */netif_stop_queue(netdev);-spin_unlock_irqrestore(&adapter->tx_lock,flags);-returnNETDEV_TX_BUSY;+smp_mb();+if(atl1c_tpd_avail(adapter,type)<tpd_req)+returnNETDEV_TX_BUSY;+netif_wake_queue(netdev);}tpd=atl1c_get_tpd(adapter,type);/* do TSO and check sum */if(atl1c_tso_csum(adapter,skb,&tpd,type)!=0){-spin_unlock_irqrestore(&adapter->tx_lock,flags);dev_kfree_skb_any(skb);returnNETDEV_TX_OK;}
Thomas
Are you using the latest atl1c code in kernel ? recently, I have updated some hw configuration.
Thanks
Xiong
-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-
owner@vger.kernel.org] On Behalf Of Thomas Meyer
Sent: Thursday, June 07, 2012 20:38
To: Jonathan Nieder
Cc: Eric Dumazet; Linux Kernel Mailing List; jcliburn@gmail.com;
chris.snook@gmail.com; netdev; Josh Boyer
Subject: Re: NETDEV WATCHDOG: eth0 (atl1c): transmit queue 0 timed out
Am Dienstag, den 05.06.2012, 19:38 -0500 schrieb Jonathan Nieder:
quoted
In February, 2012, Thomas Meyer wrote:
quoted
Am Freitag, den 24.02.2012, 20:20 +0100 schrieb Eric Dumazet:
quoted
quoted
Here is a cumulative patch to hopefuly remove the races in this
driver, could you please test it ?
[...]
quoted
just building a 3.2.7 kernel with your patch applied. I will watch
out for the warning in the next days.
Well, did it work? :)
Hi Jonathan,
no it didn't. I still get these warnings.
wiht kind regards
thomas
quoted
In suspense,
Jonathan
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body
of a message to majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html