Am Montag, 13. September 2004 01:26 schrieben Sie:
Hans-Frieder Vogt [off-list ref] :
[2.6.9-rc1-bk11 r8169 freeze on amd64]
quoted
Until somebody comes up with a proper solution for this problem, I
suggest as a work-around to introduce a parameter so that the DAC can be
simply unselected if necessary, as outlined in the patch below.
Thanks for any suggestions as to what the problem might be.
Remove the workaround, apply the attached patch and watch the oops.
Francois,
thanks for your quick response.
I applied the patch to an otherwise clean 2.6.9-rc1-bk17, but no change to
previous behaviour:
no oops (BUG_ON not triggered)! System boots up as normal, but just after I
log in on the console the system freezes, i.e., keyboard does not react any
more and the system is not accessible via network.
The time from the moment I log in to the time when the system freezes varies,
but is in the order of 5s.
There is no difference whether NAPI is enabled or not.
My .config and the boot messages are attached to this e-mail.
Hans-Frieder
If it happens in rtl8169_rx_interrupt(), you may notice that R12 is set to
0xbfc. R12 is pkt_len in rtl8169_rx_interrupt. This value is twice too
high. I have not figured why so far and I'll go to bed.
Please Cc: netdev and jgarzik@pobox.com on followup.
--
Ueimor
--
--
Hans-Frieder Vogt e-mail: hfvogt (at) arcor (dot) de
no oops (BUG_ON not triggered)! System boots up as normal, but just after I
...
log in on the console the system freezes, i.e., keyboard does not react any
more and the system is not accessible via network.
- do the keyboard leds or the magic sysrq answer (I assume you boot without X) ?
- does it make a difference if you boot with the network cable unpluged (i.e.
fine until pluged then dead when first packet comes in) ?
The time from the moment I log in to the time when the system freezes varies,
but is in the order of 5s.
First packet probably. Can you verify this point ?
There is no difference whether NAPI is enabled or not.
I will welcome lspci -vx + gcc version + objdump -S of the r8169 module.
--
Ueimor
Am Dienstag, 14. September 2004 00:02 schrieb Francois Romieu:
Hans-Frieder Vogt [off-list ref] :
[...]
quoted
no oops (BUG_ON not triggered)! System boots up as normal, but just after
I
...
quoted
log in on the console the system freezes, i.e., keyboard does not react
any more and the system is not accessible via network.
- do the keyboard leds or the magic sysrq answer (I assume you boot without
X) ?
(To be able to exclude any side effect of X, I have booted without X and I
have also removed all graphics driver related modules)
When the system freezes, the keyboard is completely dead, the LEDs do not
react any more and also the sysrq keys do not work.
- does it make a difference if you boot with the network cable
unpluged (i.e. fine until pluged then dead when first packet comes in) ?
YES!! With the network cable unplugged, the system does not freeze!
Every 10 seconds, I get now the message:
r8169: eth0: PHY reset until link up
but otherwise everything seems fine.
quoted
The time from the moment I log in to the time when the system freezes
varies, but is in the order of 5s.
First packet probably. Can you verify this point ?
I think the test with the network cable unplugged supports this assumption.
With network cable unplugged, /proc/interrupts shows 0 interrupts for the
network card, so probably the first interrupt leads to the system freeze.
quoted
There is no difference whether NAPI is enabled or not.
I will welcome lspci -vx + gcc version + objdump -S of the r8169 module.
lspci -vx and objdump -S output (gzipped) are attached, gcc version is 3.4.2
(Debian 3.4.2-2), but no visible difference with 3.4.1.
--
Ueimor
Thanks for your help, Francois.
I will put a few printks into the interrupt routine and hope to be able to
tell you more tomorrow,
Hans-Frieder
--
--
Hans-Frieder Vogt e-mail: hfvogt (at) arcor (dot) de
Francois,
I did a few tests with the r8169 driver on my x86-64 system:
with DAC enabled (as is default),
the first interrupt is a read interrupt, because the status read from
IntrStatus is 0x8001, i.e. SYSErr | RxOK.
So, a PCI error did occur, which is strange.
Because of the RxOK bit, rtl8169_rx_interrupt is called. In this routine,
tp->cur_rx is 0
tp->dirty_rx is 0
tp->RxDescArray[entry].status is 0,
which then gives a pkt_size of -4!
The system freezes somewhere in rtl8169_rx_interrupt.
When I do a
if (pkt_size < 0) break;
in rtl8169_rx_interrupt to block the errorneous pkt_size, the system still
freezes, but AFTER LEAVING the interrupt routine rtl8169_interrupt.
Same, when I introduced a shortcut to get rid of the SYSErr.
For comparison:
When I switch off DAC, then the first interrupts are
1: IntrStatus: 0x0484 (? | TxDescUnavail?), -> tx interrupt
2: IntrStatus: 0x0484, -> tx interrupt
3: IntrStatus: 0x0485, -> rx interrupt
...
Just another thought:
Of course x86-64 has the address-space that enables >4GB RAM, and x86-64
always supports DAC (as stated in include/asm-x86_64/pci.h), but I have
currently only 1GB RAM, so, strictly speaking, DAC is not really necessary.
Strange enough, the latest Realtek driver 2.2 does not even support DAC (only
the lower 32 bit of the DMA-Addresses are written to the registers).
Could it be that the Realtek driver does not support DAC for a good reason?
Anyway, I will continue searching for the problem...
Hans-Frieder
Am Dienstag, 14. September 2004 01:31 schrieb Hans-Frieder Vogt:
Am Dienstag, 14. September 2004 00:02 schrieb Francois Romieu:
quoted
Hans-Frieder Vogt [off-list ref] :
[...]
quoted
no oops (BUG_ON not triggered)! System boots up as normal, but just
after I
...
quoted
log in on the console the system freezes, i.e., keyboard does not react
any more and the system is not accessible via network.
- do the keyboard leds or the magic sysrq answer (I assume you boot
without
X) ?
(To be able to exclude any side effect of X, I have booted without X and I
have also removed all graphics driver related modules)
When the system freezes, the keyboard is completely dead, the LEDs do not
react any more and also the sysrq keys do not work.
quoted
- does it make a difference if you boot with the network cable
unpluged (i.e. fine until pluged then dead when first packet comes in) ?
YES!! With the network cable unplugged, the system does not freeze!
Every 10 seconds, I get now the message:
r8169: eth0: PHY reset until link up
but otherwise everything seems fine.
quoted
quoted
The time from the moment I log in to the time when the system freezes
varies, but is in the order of 5s.
First packet probably. Can you verify this point ?
I think the test with the network cable unplugged supports this assumption.
With network cable unplugged, /proc/interrupts shows 0 interrupts for the
network card, so probably the first interrupt leads to the system freeze.
quoted
quoted
There is no difference whether NAPI is enabled or not.
I will welcome lspci -vx + gcc version + objdump -S of the r8169 module.
lspci -vx and objdump -S output (gzipped) are attached, gcc version is
3.4.2 (Debian 3.4.2-2), but no visible difference with 3.4.1.
quoted
--
Ueimor
Thanks for your help, Francois.
I will put a few printks into the interrupt routine and hope to be able to
tell you more tomorrow,
Hans-Frieder
--
--
Hans-Frieder Vogt e-mail: hfvogt (at) arcor (dot) de
Of course x86-64 has the address-space that enables >4GB RAM, and x86-64
always supports DAC (as stated in include/asm-x86_64/pci.h), but I have
currently only 1GB RAM, so, strictly speaking, DAC is not really necessary.
Worse than that: r8169 in 2.6.9-rc[1/2] does not advertise its ability to
DMA to high memory.
Strange enough, the latest Realtek driver 2.2 does not even support DAC (only
the lower 32 bit of the DMA-Addresses are written to the registers).
Could it be that the Realtek driver does not support DAC for a good reason?
Anyway, I will continue searching for the problem...
Can you simply try the attached patch with the network cable unplugged ?
It will not fix your issue but if the result & 0x08 != 0, you can probably
stop your testing for now as it will mean "known issue".
--
Ueimor
Am Donnerstag, 16. September 2004 01:09 schrieb Francois Romieu:
Hans-Frieder Vogt [off-list ref] :
[...]
[...]
quoted
Of course x86-64 has the address-space that enables >4GB RAM, and x86-64
always supports DAC (as stated in include/asm-x86_64/pci.h), but I have
currently only 1GB RAM, so, strictly speaking, DAC is not really
necessary.
Worse than that: r8169 in 2.6.9-rc[1/2] does not advertise its ability to
DMA to high memory.
quoted
Strange enough, the latest Realtek driver 2.2 does not even support DAC
(only the lower 32 bit of the DMA-Addresses are written to the
registers). Could it be that the Realtek driver does not support DAC for
a good reason?
Anyway, I will continue searching for the problem...
Can you simply try the attached patch with the network cable unplugged ?
It will not fix your issue but if the result & 0x08 != 0, you can probably
stop your testing for now as it will mean "known issue".
--
Ueimor
r8169: eth0: Config2 = 0x10
... does not seem to be the already known issue? Anyhow, if you have more
ideas, I will be happy to test them :-)
Hans-Frieder
--
--
Hans-Frieder Vogt e-mail: hfvogt (at) arcor (dot) de
r8169: eth0: Config2 = 0x10
... does not seem to be the already known issue? Anyhow, if you have more
Typo of mine: it is. The BSD people have noticed some time ago that the DAC
of the chipset apparently does bad things when the card is inserted in a
32 bits wide slot.
Jon, if your ppc64 box offers 64 bits wide PCI slots, it would be nice if
you could ttry 2.6.9-rc2-bkX, apply
http://www.fr.zoreil.com/people/francois/misc/r8169-xx0.patch
and report the content of the "Config2" line in the logs of the kernel.
--
Ueimor
Here is the info you requested:
r8169 Gigabit Ethernet driver 1.6LK loaded
eth8: Identified chip type is 'RTL8169s/8110s'.
eth8: RTL8169 at 0xa0000000800b7000, 00:40:f4:96:fc:3f, IRQ 131
r8169: eth8: Config2 = 0x01
r8169: eth8: link up
My p630 has 64bit PCI-X slots, but my r8169 adapter is a 32bit adapter. See
lspci output below (with a 64bit PCI-X e1000 adapter as a comparison).
.....
0001:61:01.1 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet
Controller (rev 03)
Subsystem: IBM: Unknown device 0289
Flags: bus master, 66Mhz, medium devsel, latency 144, IRQ 122
Memory at cc100000 (64-bit, non-prefetchable) [size=cc000000]
Memory at cc040000 (64-bit, non-prefetchable) [size=256K]
I/O ports at 12ec00 [size=64]
Expansion ROM at 00040000 [disabled]
Capabilities: [dc] Power Management version 2
Capabilities: [e4] Capabilities: [f0] Message Signalled
Interrupts: 64bit+ Queue=0/0 Enable-
......
0002:01:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169
Gigabit Ethernet (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet
Flags: bus master, 66Mhz, medium devsel, latency 74, IRQ 131
I/O ports at 20fc00 [size=e0000000]
Memory at e0020000 (32-bit, non-prefetchable) [size=256]
Expansion ROM at 00020000 [disabled]
Capabilities: [dc] Power Management version 2
I think my AMD64 system at home has a 64bit integrated adapter, as the
performance on it is 2.5x faster (either that or r8169 and ppc don't play
well together). I will verify this when I get home.
--
Jon Mason
jdmason@us.ibm.com
Here is the info you requested:
r8169 Gigabit Ethernet driver 1.6LK loaded
eth8: Identified chip type is 'RTL8169s/8110s'.
eth8: RTL8169 at 0xa0000000800b7000, 00:40:f4:96:fc:3f, IRQ 131
r8169: eth8: Config2 = 0x01
r8169: eth8: link up
My p630 has 64bit PCI-X slots, but my r8169 adapter is a 32bit adapter.
See lspci output below (with a 64bit PCI-X e1000 adapter as a comparison).
.....
0001:61:01.1 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet
Controller (rev 03)
Subsystem: IBM: Unknown device 0289
Flags: bus master, 66Mhz, medium devsel, latency 144, IRQ 122
Memory at cc100000 (64-bit, non-prefetchable) [size=cc000000]
Memory at cc040000 (64-bit, non-prefetchable) [size=256K]
I/O ports at 12ec00 [size=64]
Expansion ROM at 00040000 [disabled]
Capabilities: [dc] Power Management version 2
Capabilities: [e4] Capabilities: [f0] Message Signalled
Interrupts: 64bit+ Queue=0/0 Enable-
......
0002:01:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169
Gigabit Ethernet (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit
Ethernet Flags: bus master, 66Mhz, medium devsel, latency 74, IRQ 131 I/O
ports at 20fc00 [size=e0000000]
Memory at e0020000 (32-bit, non-prefetchable) [size=256]
Expansion ROM at 00020000 [disabled]
Capabilities: [dc] Power Management version 2
I think my AMD64 system at home has a 64bit integrated adapter, as the
performance on it is 2.5x faster (either that or r8169 and ppc don't play
well together). I will verify this when I get home.
Well, it appears that my home system only has a 32bit adapter as well (see
below).
0000:00:0b.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169
Gigabit Ethernet (rev 10)
Subsystem: Micro-Star International Co., Ltd.: Unknown device 702c
Flags: bus master, 66Mhz, medium devsel, latency 32, IRQ 16
I/O ports at c800 [size=cff20000]
Memory at cfffb700 (32-bit, non-prefetchable) [size=256]
Expansion ROM at 00020000 [disabled]
Capabilities: [dc] Power Management version 2
Before I make any sweeping comments about the performance on ppc64, I should
probably do some more tests. I'll have to get back to you regarding that.
Would you like me to run the "Config2" patch on my amd64 system?
--
Jon Mason
jdmason@us.ibm.com
Before I make any sweeping comments about the performance on ppc64, I should
probably do some more tests. I'll have to get back to you regarding that.
Would you like me to run the "Config2" patch on my amd64 system?
Please do. If I read you correctly, 2.6.9-rc2-bkX works (more or less)
on both your ppc64 and amd64 systems, right ?
--
Ueimor
From: Andy Lutomirski <hidden> Date: 2004-09-19 21:15:31
Francois Romieu wrote:
Jon Mason [off-list ref] :
[...]
quoted
Before I make any sweeping comments about the performance on ppc64, I should
probably do some more tests. I'll have to get back to you regarding that.
Would you like me to run the "Config2" patch on my amd64 system?
Please do. If I read you correctly, 2.6.9-rc2-bkX works (more or less)
on both your ppc64 and amd64 systems, right ?
No, still broken. But I don't see any change at all in 2.6.9-rc2-bk5.
This is on amd64, with 32-bit slots (I think).
BTW, the crash is not immediate. It takes several seconds after trying
to send/recieve.
I say _trying_ because I can't ping anything. I haven't had time before
the crash to figure out what's wrong, but the device at the other end
does flash that a packet came through the wire.
FWIW, it looks like init_board is setting PCIDAC in tp->cp_cmd but that
isn't updated to the card until after the rx ring is filled in
r8169_open. This seems suspicious, since DMA memory is being allocated
possibly in >32-bit addresses but the card hasn't been told to support
that. Fixing this doesn't seem to help, though...
Turning off high DMA fixes it. Maybe it just needs to be disabled until
someone figures out what's going on.
--Andy
FWIW, it looks like init_board is setting PCIDAC in tp->cp_cmd but that
isn't updated to the card until after the rx ring is filled in
r8169_open. This seems suspicious, since DMA memory is being allocated
possibly in >32-bit addresses but the card hasn't been told to support
that. Fixing this doesn't seem to help, though...
rtl8169_hw_start() writes the CPlusCmd register before the ring descriptor
adresses are set. Can you elaborate why it would not be enough ?
Btw the r8169 driver in 2.6.9-rcX does not advertise NETIF_F_HIGHDMA: where
would a >32 bit address come from ?
Turning off high DMA fixes it. Maybe it just needs to be disabled until
someone figures out what's going on.
I am cooking a patch for it (+ check for PCI error).
As a side note, the r8169 chipset does not like DAC to be enabled on a
32bit system. I got the usual PCI error reported while trying it.
--
Ueimor
From: Andy Lutomirski <hidden> Date: 2004-09-19 23:49:32
Francois Romieu wrote:
Andy Lutomirski [off-list ref] :
[...]
quoted
FWIW, it looks like init_board is setting PCIDAC in tp->cp_cmd but that
isn't updated to the card until after the rx ring is filled in
r8169_open. This seems suspicious, since DMA memory is being allocated
possibly in >32-bit addresses but the card hasn't been told to support
that. Fixing this doesn't seem to help, though...
rtl8169_hw_start() writes the CPlusCmd register before the ring descriptor
adresses are set. Can you elaborate why it would not be enough ?
Because rtl8169_init_ring() is called before rtl8169_hw_start() in
rtl8169_open(). init_ring allocates the recieve buffers, and I'm
assuming that r8169_alloc_rx_skb() can get >32 bits.
Btw the r8169 driver in 2.6.9-rcX does not advertise NETIF_F_HIGHDMA: where
would a >32 bit address come from ?
I was looking at 2.6.9-rc2-mm1, which has (pardon hideous line wrapping):
if ((sizeof(dma_addr_t) > 4) && !pci_set_dma_mask(pdev, DMA_64BIT_MASK)) {
tp->cp_cmd |= PCIDAC;
dev->features |= NETIF_F_HIGHDMA;
^^^^^^^^^^^^^^^
} else {
rc = pci_set_dma_mask(pdev, DMA_32BIT_MASK);
if (rc < 0) {
printk(KERN_ERR PFX "DMA configuration failed.\n");
goto err_out_free_res;
}
}
But I have 512MB RAM, so this probably isn't a problem for me. I'm
running x86_64, BTW.
quoted
Turning off high DMA fixes it. Maybe it just needs to be disabled until
someone figures out what's going on.
I am cooking a patch for it (+ check for PCI error).
As a side note, the r8169 chipset does not like DAC to be enabled on a
32bit system. I got the usual PCI error reported while trying it.
Will this be ready for 2.6.9? Otherwise it would probably be better to
just disable DAC so the driver keeps working.
On an unrelated note, I just took a look at the PHY timer code. It
looks like the PHY timer will never reactivate once it shuts off, which
means that the resetting is completely deterministic and the printk
conveys no useful information. Can we just delete it (patch attached --
this computer mangles them)?
What does it do, anyway? If it is necessary for someone's card (not
mine) to acquire a gigabit link (which is what it seemed to do in old
versions of the driver), then we're _still_ broken for the case where
they drop the link and try to reacquire it. Would it make sense to
remove the reset_enable stuff in -mm for awhile and see if anyone complains?
--Andy
That sounds like a bug right there... need all the addresses set up
before we turn on stuff.
The description of the CPlusCmd in the 8169 datasheet includes a small note
which suggests that this register should be set up early.
It does not cost much to try and see if it makes a difference for DAC though.
--
Ueimor
From: Jon Mason <hidden> Date: 2004-09-20 16:06:35
Andy,
Your setup sounds very similar to mine, and I am not hitting the error. I am
running Gentoo on AMD Athlon(tm) 64 Processor 3200+ with 512MB RAM. My r8169
adapter (8110 chipset) is integrated in my MoBo. How is your setup
different?
Thanks,
--
Jon Mason
jdmason@us.ibm.com
From: Jeff Garzik <hidden> Date: 2004-09-20 18:00:03
Francois Romieu wrote:
Jeff Garzik [off-list ref] :
[...]
quoted
That sounds like a bug right there... need all the addresses set up
before we turn on stuff.
The description of the CPlusCmd in the 8169 datasheet includes a small note
which suggests that this register should be set up early.
It does not cost much to try and see if it makes a difference for DAC though.
Nothing conclusive so far.
I applied patch below on my 32bit system on top of current 2.6.9-rc2-mm1 +
patch issued yesterday + extra hack to force PCI DAC on 32 bit system.
It does not help if I force PCI DAC.
It does not change anything if PCI DAC is not enabled.
diff -puN drivers/net/r8169.c~r8169-145b-4 drivers/net/r8169.c
@@ -1494,7 +1499,6 @@ rtl8169_hw_start(struct net_device *dev)}RTL_W8(Cfg9346,Cfg9346_Unlock);-RTL_W8(ChipCmd,CmdTxEnb|CmdRxEnb);RTL_W8(EarlyTxThres,EarlyTxThld);// For gigabit rtl8169
@@ -1509,24 +1513,9 @@ rtl8169_hw_start(struct net_device *dev)RTL_W32(TxConfig,(TX_DMA_BURST<<TxDMAShift)|(InterFrameGap<<TxInterFrameGapShift));-tp->cp_cmd|=RTL_R16(CPlusCmd);-RTL_W16(CPlusCmd,tp->cp_cmd);--if(tp->mac_version==RTL_GIGA_MAC_VER_D){-dprintk(KERN_INFOPFX"Set MAC Reg C+CR Offset 0xE0. "-"Bit-3 and bit-14 MUST be 1\n");-tp->cp_cmd|=(1<<14)|PCIMulRW;-RTL_W16(CPlusCmd,tp->cp_cmd);-}-tp->cur_rx=0;-RTL_W32(TxDescStartAddrLow,((u64)tp->TxPhyAddr&DMA_32BIT_MASK));-RTL_W32(TxDescStartAddrHigh,((u64)tp->TxPhyAddr>>32));-RTL_W32(RxDescAddrLow,((u64)tp->RxPhyAddr&DMA_32BIT_MASK));-RTL_W32(RxDescAddrHigh,((u64)tp->RxPhyAddr>>32));RTL_W8(Cfg9346,Cfg9346_Lock);-udelay(10);RTL_W32(RxMissed,0);
@@ -1538,6 +1527,17 @@ rtl8169_hw_start(struct net_device *dev)/* Enable all known interrupts by setting the interrupt mask. */RTL_W16(IntrMask,rtl8169_intr_mask);+tp->cp_cmd|=RTL_R16(CPlusCmd);++if(tp->mac_version==RTL_GIGA_MAC_VER_D){+dprintk(KERN_INFOPFX"Set MAC Reg C+CR Offset 0xE0. "+"Bit-3 and bit-14 MUST be 1\n");+tp->cp_cmd|=(1<<14)|PCIMulRW;+}++RTL_W16(CPlusCmd,tp->cp_cmd);+RTL_W8(ChipCmd,CmdTxEnb|CmdRxEnb);+netif_start_queue(dev);}