Re: 2.6.20->2.6.21 - networking dies after random time

Mmm, bad news, after 4 hours of intensive network stressing, one of the 2 3com card failed with the latest fedora kernel.

Aug  6 22:31:09 loki kernel: NETDEV WATCHDOG: eth2: transmit timed out
Aug  6 22:31:09 loki kernel: eth2: transmit timed out, tx_status 00 status e601.
Aug  6 22:31:09 loki kernel:   diagnostics: net 0ccc media 8880 dma 0000003a fifo 8000
Aug  6 22:31:09 loki kernel: eth2: Interrupt posted but not delivered -- IRQ blocked by another device?
Aug  6 22:31:09 loki kernel:   Flags; bus-master 1, dirty 26085000(8) current 26085000(8)
Aug  6 22:31:09 loki kernel:   Transmit list 00000000 vs. ffff81007c807700.

Stressing eth2 by copying large files on a samba on share and eth0 by downloading big files on the internet.

Jb

* Chuck Ebbert [off-list ref] wrote:

quoted

Before, they would print:

eth0: transmit timed out, tx_status 00 status e601.
  diagnostics: net 0ccc media 8880 dma 0000003a fifo 0000
eth0: Interrupt posted but not delivered -- IRQ blocked by another device?
  Flags; bus-master 1, dirty 295757(13) current 295757(13)
  Transmit list 00000000 vs. f7150a20.
  0: @f7150200  length 80000070 status 0c010070
  1: @f71502a0  length 80000070 status 0c010070
  2: @f7150340  length 8000005c status 0c01005c

Now they just work, apparently...

could you please try the patch below? If this doesnt do the trick then i 
guess we need to revert that change.

	Ingo

------------>
(take 2)

Subject: genirq: fix simple and fasteoi irq handlers

After the "genirq: do not mask interrupts by default" patch interrupts
should be disabled not immediately upon request, but after they happen.
But, handle_simple_irq() and handle_fasteoi_irq() can skip this once or
more if an irq is just serviced (IRQ_INPROGRESS), possibly disrupting a
driver's work.

The main reason of problems here, pointing the broken patch and making
the first patch which can fix this was done by Marcin Slusarz.
Additional test patches of Thomas Gleixner and Ingo Molnar tested by
Marcin Slusarz helped to narrow possible reasons even more. Thanks.

PS: this patch fixes only one evident error here, but there could be
more places affected by above-mentioned change in irq handling.

87 more quoted lines

On 08/06/2007 04:42 PM, Jean-Baptiste Vignaud wrote:

Mmm, bad news, after 4 hours of intensive network stressing, one of the 2 3com card failed with the latest fedora kernel.

Aug  6 22:31:09 loki kernel: NETDEV WATCHDOG: eth2: transmit timed out
Aug  6 22:31:09 loki kernel: eth2: transmit timed out, tx_status 00 status e601.
Aug  6 22:31:09 loki kernel:   diagnostics: net 0ccc media 8880 dma 0000003a fifo 8000
Aug  6 22:31:09 loki kernel: eth2: Interrupt posted but not delivered -- IRQ blocked by another device?
Aug  6 22:31:09 loki kernel:   Flags; bus-master 1, dirty 26085000(8) current 26085000(8)
Aug  6 22:31:09 loki kernel:   Transmit list 00000000 vs. ffff81007c807700.

Stressing eth2 by copying large files on a samba on share and eth0 by downloading big files on the internet.

So even the full revert doesn't fix the 3Com driver, it just makes it less
likely to do that.

The other patch probably won't be any better -- I'd guess there's some
kind of IRQ handling bug in that driver.

Jean-Baptiste Vignaud wrote:

Mmm, bad news, after 4 hours of intensive network stressing, one of the 2
3com card failed with the latest fedora kernel.

Aug  6 22:31:09 loki kernel: NETDEV WATCHDOG: eth2: transmit timed out
Aug  6 22:31:09 loki kernel: eth2: transmit timed out, tx_status 00 status
e601. Aug  6 22:31:09 loki kernel:   diagnostics: net 0ccc media 8880 dma
0000003a fifo 8000 Aug  6 22:31:09 loki kernel: eth2: Interrupt posted but
not delivered -- IRQ blocked by another device? Aug  6 22:31:09 loki
kernel:   Flags; bus-master 1, dirty 26085000(8) current 26085000(8) Aug 
6 22:31:09 loki kernel:   Transmit list 00000000 vs. ffff81007c807700.

Stressing eth2 by copying large files on a samba on share and eth0 by
downloading big files on the internet.

Next time you want to stress your network you may want to try this:

  # ping 10.1 -s8 -f -l9

or

  # ping 10.1 -s8 -A > /dev/null

BTW, I mentioned this before, there maybe a BIOS irq config mismatch before 
booting the kernel.


Thanks!

--
Al

On Mon, Aug 06, 2007 at 05:19:03PM -0400, Chuck Ebbert wrote:

On 08/06/2007 04:42 PM, Jean-Baptiste Vignaud wrote:

quoted

Mmm, bad news, after 4 hours of intensive network stressing, one of the 2 3com card failed with the latest fedora kernel.

Aug  6 22:31:09 loki kernel: NETDEV WATCHDOG: eth2: transmit timed out
Aug  6 22:31:09 loki kernel: eth2: transmit timed out, tx_status 00 status e601.
Aug  6 22:31:09 loki kernel:   diagnostics: net 0ccc media 8880 dma 0000003a fifo 8000
Aug  6 22:31:09 loki kernel: eth2: Interrupt posted but not delivered -- IRQ blocked by another device?
Aug  6 22:31:09 loki kernel:   Flags; bus-master 1, dirty 26085000(8) current 26085000(8)
Aug  6 22:31:09 loki kernel:   Transmit list 00000000 vs. ffff81007c807700.

Stressing eth2 by copying large files on a samba on share and eth0 by downloading big files on the internet.

So even the full revert doesn't fix the 3Com driver, it just makes it less
likely to do that.

The other patch probably won't be any better -- I'd guess there's some
kind of IRQ handling bug in that driver.

I don't know how fast are these 3com chips regarding these 8390
described by Alan, and how are irqs shared on Jean-Baptiste's box,
but I'm surprised they could have worked sharing interrupts and
without such time outs before this change in 2.6.21. It seems some
of those older chips, because of slowness, could have transmit
problems even without irq sharing. So, IMHO, if possible, there
should be never irq sharing enabled between two (or more) drivers
using both disable_irq.

These time out problems were reported long time ago, but I think
it would be nice if this thread could at least remove these new
problems reported only after 2.6.21, which it seems is possible
now, after Marcin's diagnose: by reverting the whole 2.6.21 patch
or by this current temporary patch in 2.6.23-rc2's resend.c.
It would be nice if you could try this patch too.

BTW: Jean-Babtiste, could you send or point to you current configs?
I mean at least proc/interrupts, but with dmesg and .config it would
be even better. (I assume this last report was about the revert patch
mentioned by Chuck, not the one below your message?)

Regards,
Jarek P.