Hi all, I've identified you as people who have at some point in the past
emailed one of the Linux lists with problems with e1000 and
sk_forward_alloc. It seems to be fairly widespread, but only seems to
have appeared with recent kernel changes (after 2.6.12...)
What I need from you is a reproducible test, and some information. I
have never been able to reproduce this, and I'm trying to isolate the
problem a bit. What motherboards are you using? What seems to cause
this problem? Are you all using iptables? Are you all routing? From
the reports I assume none of you are using an 82571/2/3 (pci express)
As far as I know e1000 has the same requirement as tg3 and some others
where we have to modify the header of the skb in the case of transmits
using TSO. I don't see anywhere else that the driver modifies the skb.
Tomorrow I'll generate a patch to try a more paranoid copying of the
skb, I hope some of you can test.
To do this we have code like so in e1000_tso:
2529 if (skb_shinfo(skb)->tso_size) {
2530 if (skb_header_cloned(skb)) {
2531 err = pskb_expand_head(skb, 0, 0,
GFP_ATOMIC);
2532 if (err)
2533 return err;
2534 }
2535
2536 hdr_len = ((skb->h.raw - skb->data) +
(skb->h.th->doff << 2));
2537 mss = skb_shinfo(skb)->tso_size;
2538 if (skb->protocol == ntohs(ETH_P_IP)) {
2539 skb->nh.iph->tot_len = 0;
2540 skb->nh.iph->check = 0;
Thanks for your assistance
Jesse
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642
Hi all, I've identified you as people who have at some point in the past
emailed one of the Linux lists with problems with e1000 and
sk_forward_alloc. It seems to be fairly widespread, but only seems to
have appeared with recent kernel changes (after 2.6.12...)
What I need from you is a reproducible test, and some information. I
have never been able to reproduce this, and I'm trying to isolate the
problem a bit. What motherboards are you using? What seems to cause
this problem? Are you all using iptables? Are you all routing? From
the reports I assume none of you are using an 82571/2/3 (pci express)
As far as I know e1000 has the same requirement as tg3 and some others
where we have to modify the header of the skb in the case of transmits
using TSO. I don't see anywhere else that the driver modifies the skb.
Tomorrow I'll generate a patch to try a more paranoid copying of the
skb, I hope some of you can test.
To do this we have code like so in e1000_tso:
2529 if (skb_shinfo(skb)->tso_size) {
2530 if (skb_header_cloned(skb)) {
2531 err = pskb_expand_head(skb, 0, 0,
GFP_ATOMIC);
2532 if (err)
2533 return err;
2534 }
2535
2536 hdr_len = ((skb->h.raw - skb->data) +
(skb->h.th->doff << 2));
2537 mss = skb_shinfo(skb)->tso_size;
2538 if (skb->protocol == ntohs(ETH_P_IP)) {
2539 skb->nh.iph->tot_len = 0;
2540 skb->nh.iph->check = 0;
Thanks for your assistance
Jesse
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642
From: Phil Oester <hidden> Date: 2006-03-30 04:25:50
On Wed, Mar 29, 2006 at 06:53:57PM -0800, Brandeburg, Jesse wrote:
Hi all, I've identified you as people who have at some point in the past
emailed one of the Linux lists with problems with e1000 and
sk_forward_alloc. It seems to be fairly widespread, but only seems to
have appeared with recent kernel changes (after 2.6.12...)
What I need from you is a reproducible test, and some information. I
have never been able to reproduce this, and I'm trying to isolate the
problem a bit. What motherboards are you using? What seems to cause
this problem? Are you all using iptables? Are you all routing? From
the reports I assume none of you are using an 82571/2/3 (pci express)
Unfortunately it happens randomly, so I have no reproducible test.
Dell 1850s and 2850s here, no iptables, routing, or pci express.
lspci reports:
82541GI/PI Gigabit Ethernet Controller (rev 05)
As far as I know e1000 has the same requirement as tg3 and some others
where we have to modify the header of the skb in the case of transmits
using TSO. I don't see anywhere else that the driver modifies the skb.
Tomorrow I'll generate a patch to try a more paranoid copying of the
skb, I hope some of you can test.
I'll certainly try it as long as it doesn't blow things up :)
Phil
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
To do this we have code like so in e1000_tso:
2529 if (skb_shinfo(skb)->tso_size) {
2530 if (skb_header_cloned(skb)) {
2531 err = pskb_expand_head(skb, 0, 0,
GFP_ATOMIC);
2532 if (err)
2533 return err;
2534 }
I was wondering if that call could somehow mess up the
sk->sk_forward_alloc value later on.
But it can't, sk_forward_alloc is modified based upon the
skb->truesize value, but pskb_expand_head() does not change that.
So the things left to check in the generic networking are the
skb_shinfo() contents and ->dataref handling.
I considered whether pskb_expand_head() could corrupt the TSO
information in skb_shinfo(). But that's clearly not the case because
pskb_expand_head() explicitly copies it over:
memcpy(data + size, skb->end, sizeof(struct skb_shared_info));
And skb->end is set appropriately:
skb->end = data + size;
because skb_shinfo() is:
#define skb_shinfo(SKB) ((struct skb_shared_info *)((SKB)->end))
The only skb_shared_info that has to be explicitly setup is the
dataref, and pskb_expand_head() does that:
atomic_set(&skb_shinfo(skb)->dataref, 1);
So that all checks out.
I wonder if something funky is going on wrt. the skb_release_data()
that pskb_expand_head() does. We have that SKB_DATAREF_SHIFT thingy,
which will trigger in this case.
if (!skb->cloned ||
!atomic_sub_return(skb->nohdr ? (1 << SKB_DATAREF_SHIFT) + 1 : 1,
&skb_shinfo(skb)->dataref)) {
When we enqueue a new TCP frame we do skb_header_release() which goes:
skb->nohdr = 1;
atomic_add(1 << SKB_DATAREF_SHIFT, &skb_shinfo(skb)->dataref);
Presumably the dataref is "1" already when we get here and do this.
We will clone, the clone will set ->nohdr to 0 and will increment the
dataref.
So at this point the dataref should be:
1 /* initial reference */
+ (1 << SKB_DATAREF_SHIFT) /* from skb_header_release() */
+ 1 /* from skb_clone */
This all works out because when the clone is freed up, skb->nohdr will
be zero, so we will subtract "1" from dataref. Later when the ACK
arrives we'll free up the non-clone and this will have skb->nohdr set
to "1" and thus we'll subtract
(1 << SKB_DATAREF_SHIFT) + 1
from dataref, as per skb_release_data().
Although maybe relevant here, I just noticed that __skb_linearize()
does not clear skb->nohdr. I bet that will cause a bunch of trouble
if the original SKB had skb->nohdr set, but I cannot see how that
can occur, we only send clones out to the device and those have
skb->nohdr clear (Herbert, double check this for me please).
Luckily that thing is used rarely. Only in the dev_queue_xmit()
path when the SKB has been configured in such a way that the
transmitting device does not support so it should not be relevant
here. Also I note that __skb_linearize() is not used at all
outside of net/core/dev.c, so we should mark it static some point
soon. In fact we should do that while fixing this fringe "nohdr"
bug in __skb_linearize().
All the other dataref accesses look safe.
Herbert do you see any holes here?
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Mark Nipper <hidden> Date: 2006-03-30 08:24:41
On 29 Mar 2006, Brandeburg, Jesse wrote:
What I need from you is a reproducible test, and some information. I
have never been able to reproduce this, and I'm trying to isolate the
problem a bit. What motherboards are you using? What seems to cause
this problem? Are you all using iptables? Are you all routing? From
the reports I assume none of you are using an 82571/2/3 (pci express)
Unfortunately, my problem machine is a remote, leased
server, so I'd have to ask my provider for information on the
motherboard. I have no specific idea what causes the problem as
the assertions simply show up after the fact in my logcheck
output. I am not using iptables or routing. And I'm fairly
certain the e1000 chip is just an integrated PCI device on the
motherboard.
As far as I know e1000 has the same requirement as tg3 and some others
where we have to modify the header of the skb in the case of transmits
using TSO. I don't see anywhere else that the driver modifies the skb.
Tomorrow I'll generate a patch to try a more paranoid copying of the
skb, I hope some of you can test.
I'll be happy to test any patches you may have to narrow
down the problem. I was actually considering running tcpdump or
ethereal or some such to try to capture the event on the network
side, but this probably isn't a wise idea considering it's a
production server and I do not have hands-on access to it. A
patch which simply increased the verbosity of the event
(including counters and registers maybe?) would be preferable to
trying to capture an arbitrary amount of network traffic simply
waiting for the next time the assertion is triggered.
Sorry for the real lack of data on this end. But as I
said, any patch to help debug this is welcome.
--
Mark Nipper e-contacts:
832 Tanglewood Drive nipsy@bitgnome.net
Bryan, Texas 77802-4013 http://nipsy.bitgnome.net/
(979)575-3193 AIM/Yahoo: texasnipsy ICQ: 66971617
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GG/IT d- s++:+ a- C++$ UBL++++$ P--->+++ L+++$ !E---
W++(--) N+ o K++ w(---) O++ M V(--) PS+++(+) PE(--)
Y+ PGP t+ 5 X R tv b+++@ DI+(++) D+ G e h r++ y+(**)
------END GEEK CODE BLOCK------
---begin random quote of the moment---
I lost interest in "blade servers" when I found they didn't throw
knives at people who weren't supposed to be in your machine room.
-- Anthony de Boer
----end random quote of the moment----
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-30 08:39:37
Hello, Brandeburg, Jesse.
On 30.03.2006 06:53 you said the following:
Hi all, I've identified you as people who have at some point in the past
emailed one of the Linux lists with problems with e1000 and
sk_forward_alloc. It seems to be fairly widespread, but only seems to
have appeared with recent kernel changes (after 2.6.12...)
What I need from you is a reproducible test, and some information. I
have never been able to reproduce this, and I'm trying to isolate the
problem a bit. What motherboards are you using? What seems to cause
this problem? Are you all using iptables? Are you all routing? From
the reports I assume none of you are using an 82571/2/3 (pci express)
As far as I know e1000 has the same requirement as tg3 and some others
where we have to modify the header of the skb in the case of transmits
using TSO. I don't see anywhere else that the driver modifies the skb.
Tomorrow I'll generate a patch to try a more paranoid copying of the
skb, I hope some of you can test.
Jesse, I'd like to try your patches to help get rid of this annoying
problem. I want to say, that this problem 100% reproucible on my
hard-loading webserver based on RHEL4 with kernels 2.6.9 (rhel4
original) - 2.6.15.7 (i.e. all releases from 2.6.9 to 2.6.15.7 affected).
I have an asus 1unit server with double P4@2.8Ghz processors with
enabled HyperThreading and 3Gb RAM, but with 1Gb RAM I have the same
problem, thus it's not a RAM issue. This is really high load server,
serving about 1000-1500 http requests per second plus about 500-1000 ftp
requests per second.
dmesg, lspci -vv, iptables -nL and ip route show in attached files.
Wating for your instructions.
P.S. I use two e1000 adapters at the same time with advanced routing
like this:
[root@msk4 ~]# cat /etc/rc.local |grep ip
# This script will be executed *after* all the other init scripts.
/sbin/ip rule add from 83.102.130.174 table NEW
/sbin/ip route add default via 83.102.130.173 dev eth0 table NEW
And also I have some hardcored sysctl options like this:
net.core.somaxconn=1024
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_max_tw_buckets=720000
net.core.rmem_default=215040
net.core.rmem_max=262144
net.core.wmem_default=215040
net.core.wmem_max=262144
net.core.optmem_max=81920
net.core.netdev_max_backlog=8192
net.ipv4.neigh.default.gc_thresh1=512
net.ipv4.neigh.default.gc_thresh2=2048
net.ipv4.neigh.default.gc_thresh3=4096
net.ipv4.neigh.default.unres_qlen=64
net.ipv4.neigh.default.proxy_qlen=256
net.ipv4.tcp_rmem = 4096 131072 262144
net.ipv4.tcp_wmem = 4096 131072 262144
net.ipv4.tcp_keepalive_time=1800
net.ipv4.tcp_sack=0
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_window_scaling=0
net.ipv4.tcp_keepalive_probes=3
kernel.sem=250 32000 100 128
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
From: Johan Lundgren <hidden> Date: 2006-03-30 09:49:35
Hi,
What seems to cause this problem?
That I cannot say but the problem was fixed by removing one e1000 card
from the server (I initially had two e1000 cards installed in addition
to the two tg3 cards on the board).
Another fix was to disable TSO with ethtool.
What motherboards are you using?
Supermicro H8DAE (dual Opteron)
Are you all using iptables? Are you all routing?
Iptables yes, routing no.
none of you are using an 82571/2/3 (pci express)
Correct.
Regards,
Johan
On 3/30/06, Brandeburg, Jesse [off-list ref] wrote:
Hi all, I've identified you as people who have at some point in the past
emailed one of the Linux lists with problems with e1000 and
sk_forward_alloc. It seems to be fairly widespread, but only seems to
have appeared with recent kernel changes (after 2.6.12...)
What I need from you is a reproducible test, and some information. I
have never been able to reproduce this, and I'm trying to isolate the
problem a bit. What motherboards are you using? What seems to cause
this problem? Are you all using iptables? Are you all routing? From
the reports I assume none of you are using an 82571/2/3 (pci express)
As far as I know e1000 has the same requirement as tg3 and some others
where we have to modify the header of the skb in the case of transmits
using TSO. I don't see anywhere else that the driver modifies the skb.
Tomorrow I'll generate a patch to try a more paranoid copying of the
skb, I hope some of you can test.
To do this we have code like so in e1000_tso:
2529 if (skb_shinfo(skb)->tso_size) {
2530 if (skb_header_cloned(skb)) {
2531 err = pskb_expand_head(skb, 0, 0,
GFP_ATOMIC);
2532 if (err)
2533 return err;
2534 }
2535
2536 hdr_len = ((skb->h.raw - skb->data) +
(skb->h.th->doff << 2));
2537 mss = skb_shinfo(skb)->tso_size;
2538 if (skb->protocol == ntohs(ETH_P_IP)) {
2539 skb->nh.iph->tot_len = 0;
2540 skb->nh.iph->check = 0;
Thanks for your assistance
Jesse
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2006-03-30 09:52:45
On Wed, Mar 29, 2006 at 08:44:09PM -0800, David S. Miller wrote:
Herbert do you see any holes here?
Well I started from the beginning again, and found this. This may be
the smoking gun that we're after :)
The xmit routine is lockless but checks last_tx_tso outside the locked
section. So if a TSO packet wins a race against a non-TSO packet with
last_tx_tso == 0 then we'll have memory corruption.
Everyone, please try this patch and let us know whether the problem
goes away.
Thanks,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Boris B. Zhmurov <hidden> Date: 2006-03-30 10:02:01
Hello, Herbert Xu.
On 30.03.2006 13:52 you said the following:
On Wed, Mar 29, 2006 at 08:44:09PM -0800, David S. Miller wrote:
quoted
Herbert do you see any holes here?
Well I started from the beginning again, and found this. This may be
the smoking gun that we're after :)
The xmit routine is lockless but checks last_tx_tso outside the locked
section. So if a TSO packet wins a race against a non-TSO packet with
last_tx_tso == 0 then we'll have memory corruption.
Everyone, please try this patch and let us know whether the problem
goes away.
Thanks,
[zhmurov@builds linux-2.6.16]$ patch -p1 <
../../../SOURCES/linux-2.6.16-e1000-try-to-fix-assertion_sk_forward_alloc_failed_by_Herbert_Xu.patch
patching file drivers/net/e1000/e1000_main.c
Reversed (or previously applied) patch detected! Assume -R? [n]
Herbert, is that patch already included in 2.6.16.1?
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Krzysztof Oledzki <hidden> Date: 2006-03-30 10:27:36
On Wed, 29 Mar 2006, Brandeburg, Jesse wrote:
Hi all, I've identified you as people who have at some point in the past
emailed one of the Linux lists with problems with e1000 and
sk_forward_alloc. It seems to be fairly widespread, but only seems to
have appeared with recent kernel changes (after 2.6.12...)
What I need from you is a reproducible test, and some information. I
have never been able to reproduce this, and I'm trying to isolate the
problem a bit. What motherboards are you using?
From: Krzysztof Oledzki <hidden> Date: 2006-03-30 10:29:36
On Thu, 30 Mar 2006, Mark Nipper wrote:
On 29 Mar 2006, Brandeburg, Jesse wrote:
quoted
What I need from you is a reproducible test, and some information. I
have never been able to reproduce this, and I'm trying to isolate the
problem a bit. What motherboards are you using? What seems to cause
this problem? Are you all using iptables? Are you all routing? From
the reports I assume none of you are using an 82571/2/3 (pci express)
Unfortunately, my problem machine is a remote, leased
server, so I'd have to ask my provider for information on the
motherboard.
You can probably check this with the dmidecode tool.
Best regards,
Krzysztof Olędzki
patching file drivers/net/e1000/e1000_main.c
Reversed (or previously applied) patch detected! Assume -R? [n]
Herbert, is that patch already included in 2.6.16.1?
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-30 13:29:09
Hello, Herbert Xu.
On 30.03.2006 14:12 you said the following:
On Thu, Mar 30, 2006 at 10:02:01AM +0000, Boris B. Zhmurov wrote:
quoted
[zhmurov@builds linux-2.6.16]$ patch -p1 <
../../../SOURCES/linux-2.6.16-e1000-try-to-fix-assertion_sk_forward_alloc_failed_by_Herbert_Xu.patch
patching file drivers/net/e1000/e1000_main.c
Reversed (or previously applied) patch detected! Assume -R? [n]
Herbert, is that patch already included in 2.6.16.1?
Not really. It's just patch being silly (or too smart :)
Here it is again rediffed against 2.6.16.
Nope, with this patch the problem still exists. After 25 min. uptime
with patched kernel 2.6.16.1, I have:
Mar 30 16:30:31 msk4 kernel: KERNEL: assertion (!sk->sk_forward_alloc)
failed at net/core/stream.c (283)
Mar 30 16:30:31 msk4 kernel: KERNEL: assertion (!sk->sk_forward_alloc)
failed at net/ipv4/af_inet.c (150)
Mar 30 17:05:42 msk4 kernel: KERNEL: assertion (!sk->sk_forward_alloc)
failed at net/core/stream.c (283)
Mar 30 17:05:42 msk4 kernel: KERNEL: assertion (!sk->sk_forward_alloc)
failed at net/ipv4/af_inet.c (150)
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Phil Oester <hidden> Date: 2006-03-30 16:22:13
On 29 Mar 2006, Brandeburg, Jesse wrote:
What I need from you is a reproducible test, and some information. I
From all the reports which have come in thus far, it seems everyone
has > 1 e1000. One person even reported that removing one of the two
nics solved the problem for him. Does this help narrow down the search
area?
Phil
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Krzysztof Oledzki <hidden> Date: 2006-03-30 17:21:36
On Thu, 30 Mar 2006, Phil Oester wrote:
quoted
On 29 Mar 2006, Brandeburg, Jesse wrote:
What I need from you is a reproducible test, and some information. I
From all the reports which have come in thus far, it seems everyone
has > 1 e1000. One person even reported that removing one of the two
nics solved the problem for him. Does this help narrow down the search
area?
I have only one. Anyway, this massage happens _very_ occasionally in my
case.
Best regrads,
Krzysztof Olędzki
Hi Jesse,
More datapoints.
First of all, I don't see the problem, so this is an exclusion data point.
Machine is up 1 day, 19:02
I use 2.6.16 and I'm NBOT running at Gigabit speed.
(just couldn't get e100 cards anymore, they are not sold anymore here)
Version: vendor 00:aa:00, model 56 rev 0
Services: I'm routing, run IPsec, do firewalling/nat with iptables, do PPPoE
on this machine but all not on that interface.
The card is exposed to the local LAN interface.
[4294671.426000] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
[4294671.447000] e100: eth1: e100_probe: addr 0xe3140000, irq 10, MAC addr 00:D0:B7:XX:XX:XX
[4294671.447000] eth2: VIA Rhine II at 0xe3142000, 00:0f:ea:XX:XX:XX, IRQ 11.
[4294671.448000] eth2: MII PHY found at address 1, status 0x786d advertising 05e1 Link 41e1.
[4294671.448000] forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.49.
[4294679.125000] e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex
[4294679.165000] e100: eth1: e100_watchdog: link up, 10Mbps, half-duplex
[4294679.201000] eth2: link up, 100Mbps, full-duplex, lpa 0x41E1
lspci with info for this card.
0000:00:0c.0 Ethernet controller: Intel Corp. 82541GI/PI Gigabit Ethernet Controller
Subsystem: Intel Corp. PRO/1000 MT Desktop Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (63750ns min), Cache Line Size: 0x08 (32 bytes)
Interrupt: pin A routed to IRQ 11
Region 0: Memory at e3100000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at e3120000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at d400 [size=64]
Expansion ROM at 20100000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=2, DMOST=0, DMCRS=0, RSCEM-
Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
ip -s -s link dev eth0
1: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0e:0c:XX:XX:XX brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
648259157 1081155 0 0 0 115
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
393218241 933436 0 0 0 0
TX errors: aborted fifo window heartbeat
0 0 0 0
My config is attached, more data on request.
I can play with parameters, but cannot test patches.
Regards
Ingo Oeser
Hi Jesse,
More datapoints.
First of all, I don't see the problem, so this is an exclusion data point.
Machine is up 1 day, 19:02
I use 2.6.16 and I'm NBOT running at Gigabit speed.
If you're not running at gigabit speed, TSO is turned off
by the e1000 driver.
So that could by why you're not seeing the problem.
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: "David S. Miller" <davem@davemloft.net> Date: 2006-03-31 09:12:45
From: "Boris B. Zhmurov" <redacted>
Date: Thu, 30 Mar 2006 17:29:09 +0400
Hello, Herbert Xu.
On 30.03.2006 14:12 you said the following:
quoted
On Thu, Mar 30, 2006 at 10:02:01AM +0000, Boris B. Zhmurov wrote:
quoted
[zhmurov@builds linux-2.6.16]$ patch -p1 <
../../../SOURCES/linux-2.6.16-e1000-try-to-fix-assertion_sk_forward_alloc_failed_by_Herbert_Xu.patch
patching file drivers/net/e1000/e1000_main.c
Reversed (or previously applied) patch detected! Assume -R? [n]
Herbert, is that patch already included in 2.6.16.1?
Not really. It's just patch being silly (or too smart :)
Here it is again rediffed against 2.6.16.
Nope, with this patch the problem still exists. After 25 min. uptime
with patched kernel 2.6.16.1, I have:
Can you please double and triple check that you're really running a
kernel with the fix from Herbert applied? I make this mistake all
the time :-)
Thanks.
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: "David S. Miller" <davem@davemloft.net> Date: 2006-03-31 09:13:24
From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Thu, 30 Mar 2006 20:52:45 +1100
Well I started from the beginning again, and found this. This may be
the smoking gun that we're after :)
The xmit routine is lockless but checks last_tx_tso outside the locked
section. So if a TSO packet wins a race against a non-TSO packet with
last_tx_tso == 0 then we'll have memory corruption.
Regardless of whether this fixes the bug being discussed, this
fix should go into the e1000 driver ASAP.
Good spotting Herbert.
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2006-03-31 09:16:53
Ingo Oeser [off-list ref] wrote:
More datapoints.
First of all, I don't see the problem, so this is an exclusion data point.
Great. I think so far all the configurations that have this problem
are
e1000 + SMP + TSO
Since your machine is not SMP but has the other two things it would
indicate that this is an SMP race.
If there are anyone else out there who do have this problem and are
either using something other than e1000, have disabled TSO, or are UP,
please speak up now.
Thanks,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: "David S. Miller" <davem@davemloft.net> Date: 2006-03-31 09:35:40
From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Fri, 31 Mar 2006 20:16:53 +1100
Ingo Oeser [off-list ref] wrote:
quoted
More datapoints.
First of all, I don't see the problem, so this is an exclusion data point.
Great. I think so far all the configurations that have this problem
are
e1000 + SMP + TSO
Since your machine is not SMP but has the other two things it would
indicate that this is an SMP race.
He does not have TSO enabled, e1000 disables TSO when on a link speed
slower than gigabit.
You'll see something like the following in your logs:
e1000: eth0: e1000_watchdog_task: 10/100 speed: disabling TSO
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-31 10:16:38
Hello, David S. Miller.
On 31.03.2006 13:12 you said the following:
From: "Boris B. Zhmurov" <redacted>
Date: Thu, 30 Mar 2006 17:29:09 +0400
quoted
Hello, Herbert Xu.
On 30.03.2006 14:12 you said the following:
quoted
On Thu, Mar 30, 2006 at 10:02:01AM +0000, Boris B. Zhmurov wrote:
quoted
[zhmurov@builds linux-2.6.16]$ patch -p1 <
../../../SOURCES/linux-2.6.16-e1000-try-to-fix-assertion_sk_forward_alloc_failed_by_Herbert_Xu.patch
patching file drivers/net/e1000/e1000_main.c
Reversed (or previously applied) patch detected! Assume -R? [n]
Herbert, is that patch already included in 2.6.16.1?
Not really. It's just patch being silly (or too smart :)
Here it is again rediffed against 2.6.16.
Nope, with this patch the problem still exists. After 25 min. uptime
with patched kernel 2.6.16.1, I have:
Can you please double and triple check that you're really running a
kernel with the fix from Herbert applied? I make this mistake all
the time :-)
Thanks.
[zhmurov@builds redhat]$ rpmbuild --sign --rebuild --target=i686
/usr/src/redhat/SRPMS/kernel-2.6.16-1.14.bbel4.src.rpm
Enter pass phrase:
Pass phrase is good.
..... SKIP.....
+ echo 'Patch #295
(linux-2.6.16-e1000-try-to-fix-assertion_sk_forward_alloc_failed_by_Herbert_Xu.patch):'
Patch #295
(linux-2.6.16-e1000-try-to-fix-assertion_sk_forward_alloc_failed_by_Herbert_Xu.patch):
+ patch -p1 -s
...... SKIP .....
And xdelta tells, that e1000.ko was modified :)
P.S.
there is src.rpm of my kernel for RHEL4 and my RHEL4-clone:
ftp://builds.kernelpanic.ru/pub/linux/BBEL/updates/kernel_of_the_day/4/SRPMS/
and yum'able repo of 2.6.16 kernels _with_ Herbert's patch :), if
anybody interested:
ftp://builds.kernelpanic.ru/pub/linux/BBEL/updates/kernel_of_the_day/4/
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2006-03-31 10:39:56
On Fri, Mar 31, 2006 at 02:16:38PM +0400, Boris B. Zhmurov wrote:
And xdelta tells, that e1000.ko was modified :)
Thanks for checking again.
Anyway, it didn't take long to find another bug in the same area.
I'm afraid this driver does seem to be full of them :)
It sets last_tx_tso in between computing the number of descriptors and
calling e1000_tx_map. This is bad because e1000_tx_map gets the wrong
value for last_tx_tso and therefore may corrupt memory for every TSO
packet when the ring is almost full.
This bug exists on UP as well as SMP.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Please try this in conjunction with the previous patch.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: "David S. Miller" <davem@davemloft.net> Date: 2006-03-31 10:45:44
From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Fri, 31 Mar 2006 21:39:56 +1100
Anyway, it didn't take long to find another bug in the same area.
I'm afraid this driver does seem to be full of them :)
Indeed.
Thanks for picking through this some more Herbert. I hope we got it
this time.
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-31 10:51:15
Hello, David S. Miller.
On 31.03.2006 14:45 you said the following:
From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Fri, 31 Mar 2006 21:39:56 +1100
quoted
Anyway, it didn't take long to find another bug in the same area.
I'm afraid this driver does seem to be full of them :)
Indeed.
Thanks for picking through this some more Herbert. I hope we got it
this time.
Recompiling the kernel. I need about 2 hours to get the answer...
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Mark Nipper <hidden> Date: 2006-03-31 10:51:36
On 31 Mar 2006, David S. Miller wrote:
He does not have TSO enabled, e1000 disables TSO when on a link speed
slower than gigabit.
You'll see something like the following in your logs:
e1000: eth0: e1000_watchdog_task: 10/100 speed: disabling TSO
Um...
---
$ uname -a
Linux king 2.6.16.1 #1 SMP Thu Mar 30 06:11:33 CST 2006 i686 GNU/Linux
$ dmesg | grep -i task
e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex
$ ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
---
I know for a fact the link is 100Mbps (other than the
output from the driver itself) and I have been bitten by the
assertion.
I've been running the first patch for about the last 24
hours and have not seen any assertions yet (although they don't
occur that frequently on this server). I'll be adding the
second, most recent patch in a bit and rebooting again.
Hopefully between the two of them, that will have fixed the
problem.
--
Mark Nipper e-contacts:
832 Tanglewood Drive nipsy@bitgnome.net
Bryan, Texas 77802-4013 http://nipsy.bitgnome.net/
(979)575-3193 AIM/Yahoo: texasnipsy ICQ: 66971617
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GG/IT d- s++:+ a- C++$ UBL++++$ P--->+++ L+++$ !E---
W++(--) N+ o K++ w(---) O++ M V(--) PS+++(+) PE(--)
Y+ PGP t+ 5 X R tv b+++@ DI+(++) D+ G e h r++ y+(**)
------END GEEK CODE BLOCK------
---begin random quote of the moment---
And if I close my mind in fear, please pry it open.
----end random quote of the moment----
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-31 11:02:33
Hello, Herbert Xu.
On 31.03.2006 14:52 you said the following:
BTW, if you kept the built tree it is possible to apply the patch and
then do a make which should compile just the e1000 driver.
Cheers,
Thank's for the tip, actually I knew that :) First of, I've already
applied some other new patches from bk-commits-head. Not for the e1000
driver. And second - I didn't keep the tree, rpmbuild cleaned it up :)
That's why I'm recompiling entire kernel.
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
On Friday 31 March 2006 14:07, Boris B. Zhmurov wrote:
David, Herbert - FYI. One of my colleague confirmed, that idea "bug
reproducible only if there is more then one e1000 adapter onboard" is
true. He has a 3 servers with double intel pro 1000 adapters, and that
bug occurs. Also, he has 4 servers with double intel pro 1000 adapters
onboard, but _only one_ of them is up. And there is no such messages in
dmesg at all! Inetresting...
At least all our systems with troubles seem to have more than one e1000
though. Usually only one is active though.
We're still not 100% it is actually the E1000, it is a bit hard to reproduce
the memory corruption :/
-Andi
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-31 12:07:03
Hello, Herbert Xu.
On 31.03.2006 14:39 you said the following:
On Fri, Mar 31, 2006 at 02:16:38PM +0400, Boris B. Zhmurov wrote:
quoted
And xdelta tells, that e1000.ko was modified :)
Thanks for checking again.
Anyway, it didn't take long to find another bug in the same area.
I'm afraid this driver does seem to be full of them :)
It sets last_tx_tso in between computing the number of descriptors and
calling e1000_tx_map. This is bad because e1000_tx_map gets the wrong
value for last_tx_tso and therefore may corrupt memory for every TSO
packet when the ring is almost full.
This bug exists on UP as well as SMP.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Please try this in conjunction with the previous patch.
Cheers,
David, Herbert - FYI. One of my colleague confirmed, that idea "bug
reproducible only if there is more then one e1000 adapter onboard" is
true. He has a 3 servers with double intel pro 1000 adapters, and that
bug occurs. Also, he has 4 servers with double intel pro 1000 adapters
onboard, but _only one_ of them is up. And there is no such messages in
dmesg at all! Inetresting...
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Mark Nipper <hidden> Date: 2006-03-31 12:10:08
On 31 Mar 2006, Boris B. Zhmurov wrote:
David, Herbert - FYI. One of my colleague confirmed, that idea "bug
reproducible only if there is more then one e1000 adapter onboard" is
true. He has a 3 servers with double intel pro 1000 adapters, and that
bug occurs. Also, he has 4 servers with double intel pro 1000 adapters
onboard, but _only one_ of them is up. And there is no such messages in
dmesg at all! Inetresting...
This unfortunately is not the case. I have two e1000
interfaces but only eth1 is up and in use. And I still had
assertions. Hopefully the two already discovered problems will
fix things up for everyone though.
--
Mark Nipper e-contacts:
832 Tanglewood Drive nipsy@bitgnome.net
Bryan, Texas 77802-4013 http://nipsy.bitgnome.net/
(979)575-3193 AIM/Yahoo: texasnipsy ICQ: 66971617
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GG/IT d- s++:+ a- C++$ UBL++++$ P--->+++ L+++$ !E---
W++(--) N+ o K++ w(---) O++ M V(--) PS+++(+) PE(--)
Y+ PGP t+ 5 X R tv b+++@ DI+(++) D+ G e h r++ y+(**)
------END GEEK CODE BLOCK------
---begin random quote of the moment---
Generalizations are usually flawed by exceptions.
-- seen at http://wunderland.com/
----end random quote of the moment----
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
On Fri, Mar 31, 2006 at 01:35:40AM -0800, David S. Miller wrote:
quoted
He does not have TSO enabled, e1000 disables TSO when on a link speed
slower than gigabit.
dmesg|grep eth0
[4294671.426000] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
[4294679.125000] e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
So this theory doesn't seem to hold :-(
Indeed. But I think that only happens on PCI Express and I don't think
Ingo is using PCI Express.
Right. PCI-Express is not available in this machine.
Maybe the traffic is not enough to trigger it. External connect is just a 6MBit DSL.
Regards
Ingo Oeser
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-31 12:23:02
Hello, Mark Nipper.
On 31.03.2006 16:10 you said the following:
This unfortunately is not the case. I have two e1000
interfaces but only eth1 is up and in use. And I still had
assertions.
Can you switch to eth0? There is no problem with _eth0_, my friend says.
> And I still had
> assertions.
I'm already using kernel with second Herbert's patch. We'll see...
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-31 12:36:36
Hello, Herbert Xu.
On 31.03.2006 16:35 you said the following:
On Fri, Mar 31, 2006 at 04:23:02PM +0400, Boris B. Zhmurov wrote:
quoted
I'm already using kernel with second Herbert's patch. We'll see...
If it still fails
Not yet. But give it a time :)
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
Hello, Herbert Xu.
On 31.03.2006 14:39 you said the following:
quoted
On Fri, Mar 31, 2006 at 02:16:38PM +0400, Boris B. Zhmurov wrote:
quoted
And xdelta tells, that e1000.ko was modified :)
Thanks for checking again.
Anyway, it didn't take long to find another bug in the same area.
I'm afraid this driver does seem to be full of them :)
It sets last_tx_tso in between computing the number of descriptors and
calling e1000_tx_map. This is bad because e1000_tx_map gets the wrong
value for last_tx_tso and therefore may corrupt memory for every TSO
packet when the ring is almost full.
This bug exists on UP as well as SMP.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Please try this in conjunction with the previous patch.
Cheers,
David, Herbert - FYI. One of my colleague confirmed, that idea "bug
reproducible only if there is more then one e1000 adapter onboard" is
true. He has a 3 servers with double intel pro 1000 adapters, and that
bug occurs. Also, he has 4 servers with double intel pro 1000 adapters
onboard, but _only one_ of them is up. And there is no such messages in
dmesg at all! Inetresting...
This is not an unique thing!
Only _one_ of my 2 equal NIC get this message
NETDEV WATCHDOG: eth0: transmit timed out
e1000: eth0: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex
with the old 2.6.15.* e1000 driver!
Not the all e1000 chips ar really equal with the same P/N Number!
This can be hardware based problem, and needs workaround?
Cheers,
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
_____________ NOD32 1.584 (20031220) Információ _____________
Az üzenetet a NOD32 Antivirus System megvizsgálta.
http://www.nod32.hu
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-31 12:46:34
Hello, Boris B. Zhmurov.
On 31.03.2006 16:23 you said the following:
Hello, Mark Nipper.
On 31.03.2006 16:10 you said the following:
quoted
This unfortunately is not the case. I have two e1000
interfaces but only eth1 is up and in use. And I still had
assertions.
Can you switch to eth0? There is no problem with _eth0_, my friend says.
P.S. I have another high-load server as gateway. Same distro, same
kernels, but less memory (512Mb lowmem). eth0 up - e100, eth1 up -
e1000. No errors at all! It kinda looks like assertions happens on
systems, where the _only_ interface _eth1_ e1000 is up.
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Christiaan den Besten <hidden> Date: 2006-03-31 13:12:30
Hi !
P.S. I have another high-load server as gateway. Same distro, same kernels, but less memory (512Mb lowmem). eth0 up - e100, eth1
up - e1000. No errors at all! It kinda looks like assertions happens on systems, where the _only_ interface _eth1_ e1000 is up.
No, we have a couple gateway's asserting.
2x : Usenet feeder : Onboard eth0 and eth1 "Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)" ->
asserts (lot's of disk activity (writes) as well by the way ... ). SMP, 4Gb RAM. (2.6.14-mm2)
4x : Usenet cache : PCI-X eth0 "Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)" -> no asserts
(no disk activity). Has 2 extra onboard e1000's, but are not used (Ethernet controller: Intel Corporation 82541GI/PI Gigabit
Ethernet Controller). SMP, 2Gb RAM (2.6.15.1)
bye,
Chris
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-31 13:30:30
Hello, Christiaan den Besten.
On 31.03.2006 17:12 you said the following:
Hi !
quoted
P.S. I have another high-load server as gateway. Same distro, same
kernels, but less memory (512Mb lowmem). eth0 up - e100, eth1 up -
e1000. No errors at all! It kinda looks like assertions happens on
systems, where the _only_ interface _eth1_ e1000 is up.
No, we have a couple gateway's asserting.
Yes, my mistake :( My server asserting with eth0 and eth1 is up both...
Herbert, with your second patch still no luck. After an hour of uptime I
have assertion (!sk->sk_forward_alloc) failed at net/core/stream.c (283)
again...
Trying your debug patch.
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-31 15:08:26
Hello, Boris B. Zhmurov.
On 31.03.2006 17:30 you said the following:
Herbert, with your second patch still no luck. After an hour of uptime I
have assertion (!sk->sk_forward_alloc) failed at net/core/stream.c (283)
again...
Trying your debug patch.
Hmm... with lastest debug patch I can't see any of debug info:
e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex
e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex
e1000: eth1: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex
KERNEL: assertion (!sk->sk_forward_alloc) failed at net/core/stream.c (283)
KERNEL: assertion (!sk->sk_forward_alloc) failed at net/ipv4/af_inet.c (150)
Is it normal?
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-03-31 15:19:09
Hello, Boris B. Zhmurov.
On 31.03.2006 19:08 you said the following:
Hmm... with lastest debug patch I can't see any of debug info:
But wait a minute. Two days ago, without Herbert's patches, assertion's
errors was like this:
Mar 29 20:03:23 msk4 kernel: KERNEL: assertion (!sk->sk_forward_alloc)
failed at net/core/stream.c (279)
Mar 29 20:03:23 msk4 kernel: KERNEL: assertion (!sk->sk_forward_alloc)
failed at net/ipv4/af_inet.c (148)
and after appling patches, errors looks like this:
Mar 31 18:21:06 msk4 kernel: KERNEL: assertion (!sk->sk_forward_alloc)
failed at net/core/stream.c (283)
Mar 31 18:21:06 msk4 kernel: KERNEL: assertion (!sk->sk_forward_alloc)
failed at net/ipv4/af_inet.c (150)
stream.c (279) -> stream.c (283)
af_inet.c (148) -> af_inet.c (150)
Does it really matters?
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
That will be because the patches changed the line numbers
in the source I believe. Nothing helpful unfortunately.
--
Mark Nipper e-contacts:
832 Tanglewood Drive nipsy@bitgnome.net
Bryan, Texas 77802-4013 http://nipsy.bitgnome.net/
(979)575-3193 AIM/Yahoo: texasnipsy ICQ: 66971617
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GG/IT d- s++:+ a- C++$ UBL++++$ P--->+++ L+++$ !E---
W++(--) N+ o K++ w(---) O++ M V(--) PS+++(+) PE(--)
Y+ PGP t+ 5 X R tv b+++@ DI+(++) D+ G e h r++ y+(**)
------END GEEK CODE BLOCK------
---begin random quote of the moment---
"Whiskey-Tango-Foxtrot, over."
-- anonymous
----end random quote of the moment----
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
That will be because the patches changed the line numbers
in the source I believe. Nothing helpful unfortunately.
Ok. Anyway, as assertion is 100% repeatable on my server, I'm ready to
try any patches to get rid of this.
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
On Fri, Mar 31, 2006 at 01:35:40AM -0800, David S. Miller wrote:
quoted
He does not have TSO enabled, e1000 disables TSO when on a link speed
slower than gigabit.
dmesg|grep eth0
[4294671.426000] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
[4294679.125000] e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
So this theory doesn't seem to hold :-(
quoted
Indeed. But I think that only happens on PCI Express and I don't think
Ingo is using PCI Express.
Right. PCI-Express is not available in this machine.
First, thanks for all the responses.
6.3.9-k4 in 2.6.16 doesn't turn off TSO for 10/100, 7.0.33 in 2.6.17-pre
does, I think that will help alleviate some of the confusion.
I've been working hard to try to reproduce here, no luck so far.
Herbert's fixes are interesting and appreciated. I'm going to try to
generate tests today that will show that the bugs he's mentioned could
occur.
Jesse
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Mark Nipper <hidden> Date: 2006-04-03 21:01:23
On 31 Mar 2006, Herbert Xu wrote:
If it still fails, here is a debugging patch which should tell us
whether we need to look elsewhere.
After three days and some hours, I finally saw another
event:
---
Apr 3 13:40:53 king kernel: KERNEL: assertion (!sk->sk_forward_alloc) failed at net/core/stream.c (283)
Apr 3 13:40:53 king kernel: KERNEL: assertion (!sk->sk_forward_alloc) failed at net/ipv4/af_inet.c (150)
---
but as with the other person who reported recently, I also did
not receive the extra debugging output from Herbert's latest
patch.
Anyway, this happened with 2.6.16.1 and Herbert's three
patches. Let me know if you want me to try anything different.
I guess all of this will just go away with the latest
driver version since those of us running at 100Mbps will no
longer have TSO enabled?
--
Mark Nipper e-contacts:
832 Tanglewood Drive nipsy@bitgnome.net
Bryan, Texas 77802-4013 http://nipsy.bitgnome.net/
(979)575-3193 AIM/Yahoo: texasnipsy ICQ: 66971617
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GG/IT d- s++:+ a- C++$ UBL++++$ P--->+++ L+++$ !E---
W++(--) N+ o K++ w(---) O++ M V(--) PS+++(+) PE(--)
Y+ PGP t+ 5 X R tv b+++@ DI+(++) D+ G e h r++ y+(**)
------END GEEK CODE BLOCK------
---begin random quote of the moment---
He hoped and prayed that there wasn't an afterlife. Then he
realized there was a contradiction involved here and merely
hoped that there wasn't an afterlife.
-- Douglas Adams
----end random quote of the moment----
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Phil Oester <hidden> Date: 2006-04-03 21:39:07
On Mon, Apr 03, 2006 at 04:01:23PM -0500, Mark Nipper wrote:
After three days and some hours, I finally saw another
event:
Ack, same here. Looked hopeful, but finally saw the error today.
Phil
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-04-03 22:00:31
Hello, Phil Oester.
On 04.04.2006 01:39 you said the following:
On Mon, Apr 03, 2006 at 04:01:23PM -0500, Mark Nipper wrote:
quoted
After three days and some hours, I finally saw another
event:
Ack, same here. Looked hopeful, but finally saw the error today.
Phil
[root@msk4 ~]# dmesg |grep assertion |wc -l
176
[root@msk4 ~]# uptime
02:00:01 up 3 days, 7:31, 2 users, load average: 1.32, 0.59, 0.41
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
Some earlier had proposed that this problem appeared in 2.6.12, which was
the introduction of the 6.X series e1000 driver.
If someone would like to, can they try the 5.6.10.1 driver from the
2.6.11.X kernel?
I'd like it if you can stick with your current kernel, and if you have
trouble building the driver, go ahead and try the 5.6.10.1 driver from
http://prdownloads.sf.net/e1000
I'll also send a patch today to back-rev the xmit routine to the 5.6.10.1
state.
Jesse
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
I'll also send a patch today to back-rev the xmit routine to the 5.6.10.1
state.
I'm in a bit of a hurry, but I wanted to send these debug patches out.
Forgive me if my mailer decides to munge them.
I'd suggest trying the first one and then both together.
The first fixes up the tso function only to be like 5.6.10.1-k2.
The second builds on the first, and incorporates more of the tx changes.
I built and tested the driver with patches on 2.6.16, with pci-x adapters.
I removed some workarounds for PCIe adapters, but I dont think anyone
having this problem has a PCIe adapter anyway. I saw no TX hangs and ran
some bi-directional tests, so i think the driver should work okay. Just
warning you I did minimal testing.
*********************
e1000: transmit the old fashioned way
It seems back in the day of 2.6.11, there were no sk_forward_alloc
asserions. Forward port that transmit code to see if it fixes the issues
in today's kernel. Unfortunately it doesn't have all the bug fixes that
the current code has, but if we get transmit timeouts we can add in
workarounds appropriately.
this changes only the e1000_tso function
Signed-off-by: Jesse Brandeburg <redacted>
---
drivers/net/e1000/e1000_main.c | 37 ++++++-------------------------------
1 files changed, 6 insertions(+), 31 deletions(-)
*************************************
e1000: implement old xmit_frame
It seems back in the day of 2.6.11, there were no sk_forward_alloc
asserions. Forward port that transmit code to see if it fixes the issues
in today's kernel. Unfortunately it doesn't have all the bug fixes that
the current code has, but if we get transmit timeouts we can add in
workarounds appropriately.
this changes the e1000_xmit_frame function, and some ancilliaries
Signed-off-by: Jesse Brandeburg <redacted>
---
drivers/net/e1000/e1000_main.c | 90 ++--------------------------------------
1 files changed, 4 insertions(+), 86 deletions(-)
@@ -2967,25 +2899,11 @@ e1000_xmit_frame(struct sk_buff *skb, stfirst=tx_ring->next_to_use;-tso=e1000_tso(adapter,tx_ring,skb);-if(tso<0){-dev_kfree_skb_any(skb);-spin_unlock_irqrestore(&tx_ring->tx_lock,flags);-returnNETDEV_TX_OK;-}--if(likely(tso)){-tx_ring->last_tx_tso=1;+if(likely(e1000_tso(adapter,tx_ring,skb)))tx_flags|=E1000_TX_FLAGS_TSO;-}elseif(likely(e1000_tx_csum(adapter,tx_ring,skb)))+elseif(likely(e1000_tx_csum(adapter,tx_ring,skb)))tx_flags|=E1000_TX_FLAGS_CSUM;-/* Old method was to assume IPv4 packet by default if TSO was enabled.-*82571hardwaresupportsTSOcapabilitiesforIPv6aswell...-*nolongerassume,wemust.*/-if(likely(skb->protocol==ntohs(ETH_P_IP)))-tx_flags|=E1000_TX_FLAGS_IPV4;-e1000_tx_queue(adapter,tx_ring,tx_flags,e1000_tx_map(adapter,tx_ring,skb,first,max_per_txd,nr_frags,mss));-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
From: Boris B. Zhmurov <hidden> Date: 2006-04-06 11:49:42
Hello, Jesse Brandeburg.
On 06.04.2006 04:42 you said the following:
I built and tested the driver with patches on 2.6.16, with pci-x adapters.
I removed some workarounds for PCIe adapters, but I dont think anyone
having this problem has a PCIe adapter anyway. I saw no TX hangs and ran
some bi-directional tests, so i think the driver should work okay. Just
warning you I did minimal testing.
*********************
e1000: transmit the old fashioned way
It seems back in the day of 2.6.11, there were no sk_forward_alloc
asserions. Forward port that transmit code to see if it fixes the issues
in today's kernel. Unfortunately it doesn't have all the bug fixes that
the current code has, but if we get transmit timeouts we can add in
workarounds appropriately.
this changes only the e1000_tso function
With this one still having:
TCP: Treason uncloaked! Peer 80.72.16.78:11460/80 shrinks window
2223569515:2223569516. Repaired.
KERNEL: assertion (!sk->sk_forward_alloc) failed at net/core/stream.c (283)
KERNEL: assertion (!sk->sk_forward_alloc) failed at net/ipv4/af_inet.c (150)
e1000: implement old xmit_frame
It seems back in the day of 2.6.11, there were no sk_forward_alloc
asserions. Forward port that transmit code to see if it fixes the issues
in today's kernel. Unfortunately it doesn't have all the bug fixes that
the current code has, but if we get transmit timeouts we can add in
workarounds appropriately.
this changes the e1000_xmit_frame function, and some ancilliaries
Signed-off-by: Jesse Brandeburg <redacted>
Can't apply this one:
[zhmurov@builds linux-2.6.16]$ patch -p1 <
../../../SOURCES/linux-2.6.16-e1000-implement_old_xmit_frame.patch
patching file drivers/net/e1000/e1000_main.c
Hunk #1 succeeded at 2620 (offset -105 lines).
Hunk #2 FAILED at 2695.
Hunk #4 FAILED at 2837.
Hunk #5 FAILED at 2868.
Hunk #6 FAILED at 2899.
4 out of 6 hunks FAILED -- saving rejects to file
drivers/net/e1000/e1000_main.c.rej
--
Boris B. Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
Hello, Jesse Brandeburg.
On 06.04.2006 04:42 you said the following:
quoted
I built and tested the driver with patches on 2.6.16, with pci-x
adapters. I removed some workarounds for PCIe adapters, but I dont
think anyone having this problem has a PCIe adapter anyway. I saw no
TX hangs and ran some bi-directional tests, so i think the driver
should work okay. Just warning you I did minimal testing.
*********************
e1000: transmit the old fashioned way
It seems back in the day of 2.6.11, there were no sk_forward_alloc
asserions. Forward port that transmit code to see if it fixes the
issues
in today's kernel. Unfortunately it doesn't have all the bug fixes that
the current code has, but if we get transmit timeouts we can add in
workarounds appropriately.
this changes only the e1000_tso function
With this one still having:
TCP: Treason uncloaked! Peer 80.72.16.78:11460/80 shrinks window
2223569515:2223569516. Repaired.
KERNEL: assertion (!sk->sk_forward_alloc) failed at net/core/stream.c
(283)
KERNEL: assertion (!sk->sk_forward_alloc) failed at net/ipv4/af_inet.c
(150)
This is a very important result. It shows that the changes to the
driver to call pskb_expand_head for TSO operations are not the cause of
this problem.
We also have some new data from the last couple of days. First, I think
that this problem is likely not just E1000's fault. We have multiple
reports both in bugzilla.kernel.org and from a distro that show this
problem has occurred on (at least) tg3 driven adapters as well as e1000.
I've been able to reliably reproduce this issue in house (finally)
thanks to one of our testers. The test is using the tbench application
from the dbench package at samba.org.
on the server, start tbench_srv
on the machine you're trying to repro the issue on, start tbench 500
<server ip>, on another client start tbench 50 <server ip>
I've seen sk_forward_alloc assertions on both server and client both
running 2.6.16. We're trying to figure out where there might be a stale
pointer to an sk that accesses the data after free. something seems to
write ff ff ff ff 00 00 00 00 to memory after it is freed maybe?
It does seem that the load (the 500 threads) is important to this
failure. I've also seen a report that a memory poisoning kernel caught
the failure.
Any investigation hints for me?
quoted
e1000: implement old xmit_frame
It seems back in the day of 2.6.11, there were no sk_forward_alloc
asserions. Forward port that transmit code to see if it fixes the
issues
in today's kernel. Unfortunately it doesn't have all the bug fixes that
the current code has, but if we get transmit timeouts we can add in
workarounds appropriately.
this changes the e1000_xmit_frame function, and some ancilliaries
Signed-off-by: Jesse Brandeburg <redacted>
Can't apply this one:
[zhmurov@builds linux-2.6.16]$ patch -p1 <
../../../SOURCES/linux-2.6.16-e1000-implement_old_xmit_frame.patch
patching file drivers/net/e1000/e1000_main.c
Hunk #1 succeeded at 2620 (offset -105 lines).
Hunk #2 FAILED at 2695.
Hunk #4 FAILED at 2837.
Hunk #5 FAILED at 2868.
Hunk #6 FAILED at 2899.
4 out of 6 hunks FAILED -- saving rejects to file
drivers/net/e1000/e1000_main.c.rej
well that seems kind of lame, but I think we got the data that we needed
from the first patch.
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
We also have some new data from the last couple of days. First, I think
that this problem is likely not just E1000's fault. We have multiple
reports both in bugzilla.kernel.org and from a distro that show this
problem has occurred on (at least) tg3 driven adapters as well as e1000.
That's interesting since tg3 does not enable TSO by default for any
chip until very recent versions of the tg3 driver. And even with
those recent tg3 drivers which will enable TSO by default, it is only
done for a very specific selection of chip revisions.
Where are these reports that precisely implicate tg3?
Note that there were legitimate TSO retransmit bugs in the 2.6.14
timeframe that triggered those messages and did get fixed. And yes
some of those reports were with tg3.
We also have some new data from the last couple of days. First, I think
that this problem is likely not just E1000's fault. We have multiple
reports both in bugzilla.kernel.org and from a distro that show this
problem has occurred on (at least) tg3 driven adapters as well as e1000.
That's interesting since tg3 does not enable TSO by default for any
chip until very recent versions of the tg3 driver. And even with
those recent tg3 drivers which will enable TSO by default, it is only
done for a very specific selection of chip revisions.
Where are these reports that precisely implicate tg3?
well there was one of them here, but the tg3 bit may actually be due to
the 2.6.14 problems.
http://bugzilla.kernel.org/show_bug.cgi?id=6279
the other one is buried at a distro's bugzilla, so I can't post it.
Note that there were legitimate TSO retransmit bugs in the 2.6.14
timeframe that triggered those messages and did get fixed. And yes
some of those reports were with tg3.
I've removed the call to pskb_expand_head in e1000 (yes i know its not
quite right, but it shouldn't be fatal either) and the bug still occurs.
so its something else. I'm not giving up on this, I just want to post the
state of the investigation.
Jesse
There are 2 e1000 gigabit devices in that person's system, and
not one tg3 device.
sure, thats fine, but we just reproduced it in two seperate systems
without the e1000 driver loaded, using the instructions as mentioned in a
previous email. We used a 5704 with TSO enabled.
I'm willing to try any debug patches you might come up with.
Jesse
sure, thats fine, but we just reproduced it in two seperate systems
without the e1000 driver loaded, using the instructions as mentioned in a
previous email. We used a 5704 with TSO enabled.
sure, thats fine, but we just reproduced it in two seperate systems
without the e1000 driver loaded, using the instructions as mentioned in a
previous email. We used a 5704 with TSO enabled.
That I didn't notice, thanks for the datapoint.
you're welcome, its "hot off the presses" (we just reproduced it 5 minutes
ago)
I'm trying to isolate more of a reproduction case, I'll be sure to post if
I can find anything with more detail.
I'm trying to isolate more of a reproduction case, I'll be sure to
post if I can find anything with more detail.
I think I see the bug.
If tbench with large numbers of clients is part of what helps
reproduce it, the key might be hitting the memory limits in tcp_mem[]
and friends, or something to do with concurrent access to
sk->sk_forward_alloc.
I bet there is some race in there.
A lot of the action is in net/core/stream.c We modify
sk->sk_forward_alloc non-atomically but that should be ok
since we ought to be holding all of the correct locks when
we hit these accesses. But it is the first thing to audit.
Let's look at sk_stream_rfree() as that is invoked from SKB
freeing callbacks and is the most likely suspect for these
kinds of problems.
It is hooked up to the skb->destructor by sk_stream_set_owner_r() and
then invoked via __kfree_skb().
Nothing here takes any locks, and as stated above we modify
sk->sk_forward_alloc non-atomically, and this is therefore the bug.
Shit.
I'll think of how to fix this in the least invasive manner. I also
want to search the changelog history to see if this race was always
present or if it was "introduced".
Making sk->sk_forward_alloc an atomic_t would be incredibly expensive
so I'll try to find a way to avoid that. We may be able to just do
a bh_lock_sock()/bh_unlock_sock() around the body of sk_stream_rfree()
to fix this.