From: Nate Case <hidden> Date: 2008-07-21 22:57:29
On Fri, 2008-07-18 at 14:10 +0200, Sebastian Siewior wrote:
Commit 35b5f6b1a aka [PHYLIB: Locking fixes for PHY I/O potentially sleeping]
changed the phydev->lock from spinlock into a mutex. Now, the following
code path got triggered while NFS was unavailable:
Hmm.. I'm not sure what the best solution is to this. Make the
stop_gfar() call happen in a workqueue, and make a similar change to
ucc_geth, fec_mpc52xx, and fs_enet? Modify phy_stop() to do the work in
a workqueue conditionally if in interrupt context? Between these two
I'd lean toward the latter.
Does anyone have any better ideas?
--
Nate Case [off-list ref]
From: Wolfram Sang <hidden> Date: 2008-07-22 07:54:42
Hi,
On Fri, Jul 18, 2008 at 02:10:08PM +0200, Sebastian Siewior wrote:
Commit 35b5f6b1a aka [PHYLIB: Locking fixes for PHY I/O potentially sleeping]
changed the phydev->lock from spinlock into a mutex. Now, the following
code path got triggered while NFS was unavailable:
[...]
I found out that the same code path may be trigger in
- drivers/net/ucc_geth.c
- drivers/net/fec_mpc52xx.c
Recently, I described a (I think) similar problem:
(http://ozlabs.org/pipermail/linuxppc-dev/2008-July/059686.html)
===
Hello,
today, I was debugging a kernel crash on a board with a MPC5200B using
2.6.26-rc9. I found the following code in drivers/net/fec_mpc52xx.c:
static irqreturn_t mpc52xx_fec_interrupt(int irq, void *dev_id)
{
[...]
/* on fifo error, soft-reset fec */
if (ievent & (FEC_IEVENT_RFIFO_ERROR | FEC_IEVENT_XFIFO_ERROR)) {
if (net_ratelimit() && (ievent & FEC_IEVENT_RFIFO_ERROR))
dev_warn(&dev->dev, "FEC_IEVENT_RFIFO_ERROR\n");
if (net_ratelimit() && (ievent & FEC_IEVENT_XFIFO_ERROR))
dev_warn(&dev->dev, "FEC_IEVENT_XFIFO_ERROR\n");
mpc52xx_fec_reset(dev);
netif_wake_queue(dev);
return IRQ_HANDLED;
}
[...]
}
Calling mpc52xx_fec_reset() from interrupt context is bad, at least
because
a) it calls phy_write, which contains BUG_ON(in_interrupt())
b) it calls mpc52xx_fec_hw_init, which has a delay-loop to check
if the reset was successful (1..50 us)
I assume the proper thing to do is to set a flag in the ISR and handle
the soft reset later in some other context. Having never dealt with the
network core and its drivers so far, I am not sure which place would be
the right one to perform the soft reset. To not make things worse, I
hope people with more insight to network stuff can deliver a suitable
solution to this problem.
All the best,
Wolfram
===
--
Dipl.-Ing. Wolfram Sang | http://www.pengutronix.de
Pengutronix - Linux Solutions for Science and Industry
From: Sebastian Siewior <hidden> Date: 2008-07-22 20:59:25
* Nate Case | 2008-07-21 17:57:08 [-0500]:
On Fri, 2008-07-18 at 14:10 +0200, Sebastian Siewior wrote:
quoted
Commit 35b5f6b1a aka [PHYLIB: Locking fixes for PHY I/O potentially sleeping]
changed the phydev->lock from spinlock into a mutex. Now, the following
code path got triggered while NFS was unavailable:
Hmm.. I'm not sure what the best solution is to this. Make the
stop_gfar() call happen in a workqueue, and make a similar change to
ucc_geth, fec_mpc52xx, and fs_enet? Modify phy_stop() to do the work in
a workqueue conditionally if in interrupt context? Between these two
I'd lean toward the latter.
Does anyone have any better ideas?
If I look at tg3.c than exactly this is done. Others call it only on
close(). I guess this depends very much on driver's logic :)
If nobody minds, than I would assume that tg3.c is a good example and I
would move the timout path into a workqueu.
Sebastian
From: Sebastian Siewior <hidden> Date: 2008-07-23 20:03:55
From: Sebastian Siewior <bigeasy@linutronix.de>
I got the following backtrace while network was unavailble:
|NETDEV WATCHDOG: eth0: transmit timed out
|BUG: sleeping function called from invalid context at /home/bigeasy/git/linux-2.6-powerpc/kernel/mutex.c:87
|in_atomic():1, irqs_disabled():0
|Call Trace:
|[c0383d90] [c0006dd8] show_stack+0x48/0x184 (unreliable)
|[c0383db0] [c001e938] __might_sleep+0xe0/0xf4
|[c0383dc0] [c025a43c] mutex_lock+0x24/0x3c
|[c0383de0] [c019005c] phy_stop+0x20/0x70
|[c0383df0] [c018d4ec] stop_gfar+0x28/0xf4
|[c0383e10] [c018e8c4] gfar_timeout+0x30/0x60
|[c0383e20] [c01fe7c0] dev_watchdog+0xa8/0x144
|[c0383e30] [c002f93c] run_timer_softirq+0x148/0x1c8
|[c0383e60] [c002b084] __do_softirq+0x5c/0xc4
|[c0383e80] [c00046fc] do_softirq+0x3c/0x54
|[c0383e90] [c002ac60] irq_exit+0x3c/0x5c
|[c0383ea0] [c000b378] timer_interrupt+0xe0/0xf8
|[c0383ec0] [c000e5ac] ret_from_except+0x0/0x18
|[c0383f80] [c000804c] cpu_idle+0xcc/0xdc
|[c0383fa0] [c025c07c] etext+0x7c/0x90
|[c0383fc0] [c0338960] start_kernel+0x294/0x2a8
|[c0383ff0] [c00003dc] skpinv+0x304/0x340
|------------[ cut here ]------------
The phylock was once a spinlock but got changed into a mutex via
commit 35b5f6b1a aka [PHYLIB: Locking fixes for PHY I/O potentially sleeping]
Signed-off-by: Sebastian Siewior <bigeasy@linutronix.de>
---
bug report @ http://marc.info/?l=linux-netdev&m=121638307116389&w=2
I moved it into a workqueue, this is what tg3 does.
I would convert the other three drivers unless $dude suggests a better
method or somebody else takes care....
drivers/net/gianfar.c | 22 ++++++++++++++++++----
drivers/net/gianfar.h | 2 ++
2 files changed, 20 insertions(+), 4 deletions(-)
@@ -216,6 +217,7 @@ static int gfar_probe(struct platform_device *pdev)spin_lock_init(&priv->txlock);spin_lock_init(&priv->rxlock);+INIT_WORK(&priv->reset_task,gfar_reset_task);platform_set_drvdata(pdev,dev);
@@ -1132,6 +1134,7 @@ static int gfar_close(struct net_device *dev)napi_disable(&priv->napi);#endif+cancel_work_sync(&priv->reset_task);stop_gfar(dev);/* Disconnect from the PHY */
@@ -1246,13 +1249,16 @@ static int gfar_change_mtu(struct net_device *dev, int new_mtu)return0;}-/* gfar_timeout gets called when a packet has not been+/* gfar_reset_task gets scheduled when a packet has not been*transmittedafterasetamountoftime.*Fornow,assumethatclearingoutallthestructures,and-*startingoverwillfixtheproblem.*/-staticvoidgfar_timeout(structnet_device*dev)+*startingoverwillfixtheproblem.+*/+staticvoidgfar_reset_task(structwork_struct*work){-dev->stats.tx_errors++;+structgfar_private*priv=container_of(work,structgfar_private,+reset_task);+structnet_device*dev=priv->dev;if(dev->flags&IFF_UP){stop_gfar(dev);
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2008-07-23 22:13:08
On Mon, 2008-07-21 at 17:57 -0500, Nate Case wrote:
On Fri, 2008-07-18 at 14:10 +0200, Sebastian Siewior wrote:
quoted
Commit 35b5f6b1a aka [PHYLIB: Locking fixes for PHY I/O potentially sleeping]
changed the phydev->lock from spinlock into a mutex. Now, the following
code path got triggered while NFS was unavailable:
Hmm.. I'm not sure what the best solution is to this. Make the
stop_gfar() call happen in a workqueue, and make a similar change to
ucc_geth, fec_mpc52xx, and fs_enet? Modify phy_stop() to do the work in
a workqueue conditionally if in interrupt context? Between these two
I'd lean toward the latter.
Does anyone have any better ideas?
From: Sebastian Siewior <hidden> Date: 2008-07-24 07:27:53
* Benjamin Herrenschmidt | 2008-07-24 08:12:48 [+1000]:
On Mon, 2008-07-21 at 17:57 -0500, Nate Case wrote:
quoted
On Fri, 2008-07-18 at 14:10 +0200, Sebastian Siewior wrote:
quoted
Commit 35b5f6b1a aka [PHYLIB: Locking fixes for PHY I/O potentially sleeping]
changed the phydev->lock from spinlock into a mutex. Now, the following
code path got triggered while NFS was unavailable:
Hmm.. I'm not sure what the best solution is to this. Make the
stop_gfar() call happen in a workqueue, and make a similar change to
ucc_geth, fec_mpc52xx, and fs_enet? Modify phy_stop() to do the work in
a workqueue conditionally if in interrupt context? Between these two
I'd lean toward the latter.
Does anyone have any better ideas?
From: Nate Case <hidden> Date: 2008-07-25 14:16:32
On Wed, 2008-07-23 at 22:03 +0200, Sebastian Siewior wrote:
I moved it into a workqueue, this is what tg3 does.
I would convert the other three drivers unless $dude suggests a better
method or somebody else takes care....
drivers/net/gianfar.c | 22 ++++++++++++++++++----
drivers/net/gianfar.h | 2 ++
2 files changed, 20 insertions(+), 4 deletions(-)
This looks good to me.
--
Nate Case [off-list ref]