From: Ben Hutchings <hidden> Date: 2012-01-29 04:32:34
[Trying a different address.]
Denis,
It looks like you were working on sundance for a while; are you still
interested in it?
Mike reported that:
Network traffic on my D-Link DFE-580TX card results in a transmit
queue timeout and gives endless resets after that untill the interface
is brought down.
The amount of traffic required to generate the error seems to vary but
sooner rather then later it will occur.
I'll check this out. After kernel.org was cracked I've missed
@kernel.org mail account.
On 1/29/12, Ben Hutchings [off-list ref] wrote:
[Trying a different address.]
Denis,
It looks like you were working on sundance for a while; are you still
interested in it?
Mike reported that:
quoted
Network traffic on my D-Link DFE-580TX card results in a transmit
queue timeout and gives endless resets after that untill the interface
is brought down.
The amount of traffic required to generate the error seems to vary but
sooner rather then later it will occur.
From: Eric Dumazet <hidden> Date: 2012-01-30 10:14:07
Le lundi 30 janvier 2012 à 12:51 +0300, Denis Kirjanov a écrit :
I'll check this out. After kernel.org was cracked I've missed
@kernel.org mail account.
At first glance, start_tx() is racy against TX completion.
It does :
if (np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 1 &&
!netif_queue_stopped(dev)) {
/* do nothing */
} else {
netif_stop_queue (dev);
}
So it can call netif_stop_queue() while TX completion handler did a
cleanup of all queued packets right before.
Note intr_handler() doesnt hold the queue spinlock when it does :
if (netif_queue_stopped(dev) &&
np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 4) {
/* The ring is no longer full, clear busy flag. */
netif_wake_queue (dev);
}
--
To UNSUBSCRIBE, email to debian-bugs-dist-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
From: Eric Dumazet <hidden> Date: 2012-01-30 10:36:08
Le lundi 30 janvier 2012 à 11:14 +0100, Eric Dumazet a écrit :
Le lundi 30 janvier 2012 à 12:51 +0300, Denis Kirjanov a écrit :
quoted
I'll check this out. After kernel.org was cracked I've missed
@kernel.org mail account.
At first glance, start_tx() is racy against TX completion.
It does :
if (np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 1 &&
!netif_queue_stopped(dev)) {
/* do nothing */
} else {
netif_stop_queue (dev);
}
So it can call netif_stop_queue() while TX completion handler did a
cleanup of all queued packets right before.
Note intr_handler() doesnt hold the queue spinlock when it does :
if (netif_queue_stopped(dev) &&
np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 4) {
/* The ring is no longer full, clear busy flag. */
netif_wake_queue (dev);
}
So I would try following patch :
drivers/net/ethernet/dlink/sundance.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
From: Ben Hutchings <hidden> Date: 2012-01-30 14:05:19
On Mon, 2012-01-30 at 11:14 +0100, Eric Dumazet wrote:
Le lundi 30 janvier 2012 à 12:51 +0300, Denis Kirjanov a écrit :
quoted
I'll check this out. After kernel.org was cracked I've missed
@kernel.org mail account.
At first glance, start_tx() is racy against TX completion.
It does :
if (np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 1 &&
!netif_queue_stopped(dev)) {
/* do nothing */
} else {
netif_stop_queue (dev);
}
So it can call netif_stop_queue() while TX completion handler did a
cleanup of all queued packets right before.
Yes, I spotted that. But no descriptors are pushed to the hardware
here; that's done in the driver's TX tasklet. Although... maybe that
can run immediately when scheduled from here? I've never had to deal
with tasklets so I really don't know their semantics.
Ben.
Note intr_handler() doesnt hold the queue spinlock when it does :
if (netif_queue_stopped(dev) &&
np->cur_tx - np->dirty_tx < TX_QUEUE_LEN - 4) {
/* The ring is no longer full, clear busy flag. */
netif_wake_queue (dev);
}
--
Ben Hutchings
Lowery's Law:
If it jams, force it. If it breaks, it needed replacing anyway.
From: Eric Dumazet <hidden> Date: 2012-01-30 14:29:00
Le lundi 30 janvier 2012 à 14:05 +0000, Ben Hutchings a écrit :
Yes, I spotted that. But no descriptors are pushed to the hardware
here; that's done in the driver's TX tasklet. Although... maybe that
can run immediately when scheduled from here? I've never had to deal
with tasklets so I really don't know their semantics.
From: Ben Hutchings <hidden> Date: 2012-01-30 14:41:33
On Mon, 2012-01-30 at 15:28 +0100, Eric Dumazet wrote:
Le lundi 30 janvier 2012 à 14:05 +0000, Ben Hutchings a écrit :
quoted
Yes, I spotted that. But no descriptors are pushed to the hardware
here; that's done in the driver's TX tasklet. Although... maybe that
can run immediately when scheduled from here? I've never had to deal
with tasklets so I really don't know their semantics.
Thats probable on SMP ...
The bug report is for a UP system running a kernel built with
SMP-alternatives.
Ben.
--
Ben Hutchings
Lowery's Law:
If it jams, force it. If it breaks, it needed replacing anyway.
From: Eric Dumazet <hidden> Date: 2012-01-30 14:57:40
Le lundi 30 janvier 2012 à 14:41 +0000, Ben Hutchings a écrit :
On Mon, 2012-01-30 at 15:28 +0100, Eric Dumazet wrote:
quoted
Le lundi 30 janvier 2012 à 14:05 +0000, Ben Hutchings a écrit :
quoted
Yes, I spotted that. But no descriptors are pushed to the hardware
here; that's done in the driver's TX tasklet. Although... maybe that
can run immediately when scheduled from here? I've never had to deal
with tasklets so I really don't know their semantics.
Thats probable on SMP ...
The bug report is for a UP system running a kernel built with
SMP-alternatives.
Hmm, TX _completion_ is not run from tasklet but hardware IRQ, this is
why I added the spin_lock_irqsave().
Tasklet fires the TX, but hardware IRQ does the TX completion part.
This driver is ... interesting :)
From: Eric Dumazet <hidden> Date: 2012-01-30 15:21:44
Le lundi 30 janvier 2012 à 15:57 +0100, Eric Dumazet a écrit :
Hmm, TX _completion_ is not run from tasklet but hardware IRQ, this is
why I added the spin_lock_irqsave().
Tasklet fires the TX, but hardware IRQ does the TX completion part.
This driver is ... interesting :)
Oh well, we also must make sure we held np->lock in TX completion when
doing our test to eventually call netif_wake_queue(), I missed it was
released too early.
here is a more complete patch.
Oh well, we also must make sure we held np->lock in TX completion when
doing our test to eventually call netif_wake_queue(), I missed it was
released too early.
here is a more complete patch.
I applied the patch, recompiled the module, loaded it into the kernel and started testing traffic on the interface with the following result :
[ 1124.008030] ------------[ cut here ]------------
[ 1124.008101] WARNING: at /build/buildd-linux-2.6_3.2.1-2-i386-4wAPNj/linux-2.6-3.2.1/debian/build/source_i386_none/net/sched/sch_generic.c:255 dev_watchdog+0xb1/0x104()
[ 1124.008201] Hardware name:
[ 1124.008252] NETDEV WATCHDOG: eth1 (sundance): transmit queue 0 timed out
[ 1124.008309] Modules linked in: sundance(O) p4_clockmod cpufreq_powersave cpufreq_userspace cpufreq_conservative cpufreq_stats speedstep_lib mperf fuse w83627ehf hwmon_vid coretemp loop ohci_hcd snd_intel8x0 snd_ac97_codec ehci_hcd ac97_bus snd_pcm usbcore snd_seq snd_timer snd_seq_device shpchp psmouse snd sis900 pci_hotplug serio_raw pcspkr mii evdev soundcore parport_pc snd_page_alloc parport processor tpm_tis tpm tpm_bios thermal_sys button usb_common ext3 jbd mbcache sd_mod crc_t10dif ata_generic sata_sis pata_sis libata scsi_mod [last unloaded: sundance]
[ 1124.010147] Pid: 5122, comm: gnome-terminal Tainted: G O 3.2.0-1-686-pae #1
[ 1124.010219] Call Trace:
[ 1124.010286] [<c1038280>] ? warn_slowpath_common+0x68/0x79
[ 1124.010344] [<c1229e38>] ? dev_watchdog+0xb1/0x104
[ 1124.010399] [<c10382f9>] ? warn_slowpath_fmt+0x29/0x2d
[ 1124.010455] [<c1229e38>] ? dev_watchdog+0xb1/0x104
[ 1124.010511] [<c103ccb5>] ? local_bh_enable+0x2/0x2
[ 1124.010567] [<c1041e78>] ? run_timer_softirq+0x150/0x1f3
[ 1124.010622] [<c1229d87>] ? netif_tx_unlock+0x3a/0x3a
[ 1124.010678] [<c103ccb5>] ? local_bh_enable+0x2/0x2
[ 1124.010733] [<c103cd49>] ? __do_softirq+0x94/0x12f
[ 1124.010788] [<c103ccb5>] ? local_bh_enable+0x2/0x2
[ 1124.010841] <IRQ> [<c103cf3a>] ? irq_exit+0x32/0x80
[ 1124.010931] [<c101e6f4>] ? smp_apic_timer_interrupt+0x5b/0x65
[ 1124.012339] [<c12b9b11>] ? apic_timer_interrupt+0x31/0x38
[ 1124.012397] [<c12b007b>] ? set_cpu_sibling_map+0x200/0x250
[ 1124.012452] ---[ end trace d55b57d11770d7d5 ]---
After this the same repeat of transmit timeouts (as posted earlier) in the log untill I down the interface.
Mike.
From: Jonathan Nieder <hidden> Date: 2012-03-16 22:04:31
Hi again,
Mike . wrote:
quoted
Oh well, we also must make sure we held np->lock in TX completion when
doing our test to eventually call netif_wake_queue(), I missed it was
released too early.
here is a more complete patch.
I applied the patch, recompiled the module, loaded it into the kernel and
started testing traffic on the interface with the following result :
[ 1124.008030] ------------[ cut here ]------------
[ 1124.008101] WARNING: at /build/buildd-linux-2.6_3.2.1-2-i386-4wAPNj/linux-2.6-3.2.1/debian/build/source_i386_none/net/sched/sch_generic.c:255 dev_watchdog+0xb1/0x104()
[ 1124.008201] Hardware name:
[ 1124.008252] NETDEV WATCHDOG: eth1 (sundance): transmit queue 0 timed out
[...]
After this the same repeat of transmit timeouts (as posted earlier) in the
log untill I down the interface.
Thanks. I assume current 3.3 release candidates behave the same way.
Based on [2], it looks like v2.6.25-rc9~99^2~24 ([NET]: Add preemption
point in qdisc_run, 2008-03-28) made this easier to trip.
As for the next step: I'd suggest posting a summary of the symptoms,
which kernel versions you have tested, and a link to [1] at
http://bugzilla.kernel.org/, product Drivers, component Network, and
letting us know the bug number so we can track it without forgetting
what has already been learned.
Hope that helps,
Jonathan
[1] http://thread.gmane.org/gmane.linux.network/219101
From: Jonathan Nieder <hidden> Date: 2012-03-16 22:11:36
Jonathan Nieder wrote:
Thanks. I assume current 3.3 release candidates behave the same way.
Based on [2], it looks like v2.6.25-rc9~99^2~24 ([NET]: Add preemption
point in qdisc_run, 2008-03-28) made this easier to trip.
As for the next step: I'd suggest posting a summary of the symptoms,
which kernel versions you have tested, and a link to [1] at
http://bugzilla.kernel.org/
Here's the missing footnote[2]. Sorry for the noise.
[...]
Date: Fri, 16 Mar 2012 17:04:13 -0500
From: jrnieder@gmail.com
To: mike-bugreport@hotmail.com
CC: eric.dumazet@gmail.com; ben@decadent.org.uk; kirjanov@gmail.com; netdev@vger.kernel.org; benoit.mortier@opensides.be; herbert@gondor.apana.org.au
Subject: Re: Sundance network driver (D-Link DFE-580TX) timeouts rendering interface unusable
Hi again,
Mike . wrote:
quoted
quoted
Oh well, we also must make sure we held np->lock in TX completion when
doing our test to eventually call netif_wake_queue(), I missed it was
released too early.
here is a more complete patch.
I applied the patch, recompiled the module, loaded it into the kernel and
started testing traffic on the interface with the following result :
[ 1124.008030] ------------[ cut here ]------------
[ 1124.008101] WARNING: at /build/buildd-linux-2.6_3.2.1-2-i386-4wAPNj/linux-2.6-3.2.1/debian/build/source_i386_none/net/sched/sch_generic.c:255 dev_watchdog+0xb1/0x104()
[ 1124.008201] Hardware name:
[ 1124.008252] NETDEV WATCHDOG: eth1 (sundance): transmit queue 0 timed out
[...]
quoted
After this the same repeat of transmit timeouts (as posted earlier) in the
log untill I down the interface.
Thanks. I assume current 3.3 release candidates behave the same way.
Based on [2], it looks like v2.6.25-rc9~99^2~24 ([NET]: Add preemption
point in qdisc_run, 2008-03-28) made this easier to trip.
As for the next step: I'd suggest posting a summary of the symptoms,
which kernel versions you have tested, and a link to [1] at
http://bugzilla.kernel.org/, product Drivers, component Network, and
letting us know the bug number so we can track it without forgetting
what has already been learned.
Hope that helps,
Jonathan
[1] http://thread.gmane.org/gmane.linux.network/219101