Re: 2.6.10-rc1-mm4 -1 EAGAIN after allocation failure was: Re: Kernel 2.6.9 Multiple Page Allocation Failures

Added netdev to Cc.  Full cite.

Stefan Schmidt [off-list ref] wrote:

Alright, i got a funny thing here that i suspect to be an(other?) vm issue:

We are running a third-party closed source software which handles many tcp
sessions and reads and writes that to/from several disks/partitions.
With 2.6.10-rc1-mm4 it is the first time we notice that, right after the kernel
throws a swapper: page allocation error thread (just like the ones you already
know), the interrupt rate, connection count and traffic decreases subsequently.

Here is part of a vmstat 10:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  11312  19404    268 1896896    0    0  1091 17578 25463  1225  7 38 37 18
 0  0  11312  26372    180 1892836    0    0  1182 21433 25576  1216  7 38 31 24
 1  2  11308  23784    608 1890168    0    0  1252 20667 25532  1243  7 40 24 29
 0  2  11308  23304    428 1890552    0    0  1174 20363 25948  1332  7 40 32 21
 1  1  11304  18496    444 1893328    0    0  1630 20506 25840  1322  7 38 30 26
 1  1  11304   8712    232 1905508    0    0  1528 19662 26245  1305  7 40 25 28
 1  0  11304  18952    180 1894000    0    0  1595 19680 26275  1215  7 38 27 28
 1  0  11304  22404    132 1896632    0    0   369 17724 24072  1045  7 37 49  7
 1  0  11304  23956    492 1899876    0    0   504 19609 20829  1151  9 34 49  7
 1  0  11304  25380    460 1908340    0    0   424 17983 16964   927  9 28 55  8
 1  0  11304  18244    464 1922140    0    0   309 14431 13417   836 10 27 60  3
 0  0  11304  17720    472 1928388    0    0   224 11868 9933   607 11 23 63  3
 1  0  11304  25720    476 1924440    0    0   133  7663 6780   504 10 20 68  2
 1  0  11304  24156    488 1928168    0    0   107  6244 5011   315  8 18 73  1
 0  0  11304  19544    712 1934268    0    0    76  3191 4464   299  8 18 73  1
 0  0  11304  19248    728 1936564    0    0    23  1802 4002   249  7 17 76  0
 1  0  11304  27092    736 1929892    0    0    16  1336 3655   284  6 16 78  0
 0  0  11304  26472    752 1931984    0    0    19  1508 3408   248  5 16 78  1
 0  0  11304  19000    768 1940944    0    0    20  1398 3195   226  5 14 81  0
 1  0  11304  21460    776 1938896    0    0    14  1084 3057   241  5 14 82  0
 0  0  11304  26268    848 1934608    0    0    12   927 2906   218  5 13 82  0
 1  1  11304  22076    900 1939860    0    0    18   679 2897   215  5 11 84  1
 0  0  11304  25880    952 1936748    0    0    17   653 2713   251  4 13 82  1
 0  0  11304  20436    976 1942368    0    0     8  1117 2703   229  5 11 83  1
...

strace shows:
01:38:50.316041 gettimeofday({1100047130, 316054}, NULL) = 0
01:38:50.316188 poll([{fd=5671, events=POLLIN}, {fd=2727, events=POLLIN}, {fd=6663, events=POLLIN}, {fd=197, events=POLLIN}, {fd=3978, events=POLLIN}, {fd=779, events=POLLIN}, ...{line continues like this}...
...
01:38:50.328056 accept(5, 0xbffd4ab8, [16]) = -1 EAGAIN (Resource temporarily unavailable) ...{an awful lot of these}...
...
01:38:50.329585 futex(0xaf1a698, FUTEX_WAIT, 92828, {0, 9964000}) = -1 ETIMEDOUT (Connection timed out) ...{some of these}...
...
Application says:
"n.n.n.n:p Client closed connection in body read"

To me it seems like suddently all those open sockets are suddenly 'temporarily
unavailable' to the application and so the connections time out.
I have not (yet?) seen this behaviour on 2.6.9, 2.6.9-mm1, 2.6.10-rc1-bk12 or
2.6.10-rc1-mm3.
I am able to reproduce the behaviour if under the same load iptraf or
tethereal are started. (First thought it might be because of the promisc mode.)

This seems to be what _might_ have triggered this although it was logged
happened 5m earlier than the traffic decay:

 printk: 36 messages suppressed.
 swapper: page allocation failure. order:0, mode:0x20
  [__alloc_pages+525/912] __alloc_pages+0x20d/0x390
  [__get_free_pages+24/48] __get_free_pages+0x18/0x30
  [kmem_getpages+24/192] kmem_getpages+0x18/0xc0
  [cache_grow+157/304] cache_grow+0x9d/0x130
  [cache_alloc_refill+380/576] cache_alloc_refill+0x17c/0x240
  [__kmalloc+122/144] __kmalloc+0x7a/0x90
  [alloc_skb+50/208] alloc_skb+0x32/0xd0
  [tg3_alloc_rx_skb+112/304] tg3_alloc_rx_skb+0x70/0x130
  [tg3_rx+518/944] tg3_rx+0x206/0x3b0
  [tg3_poll+139/336] tg3_poll+0x8b/0x150
  [net_rx_action+125/288] net_rx_action+0x7d/0x120
  [__do_softirq+184/208] __do_softirq+0xb8/0xd0
  [do_softirq+45/48] do_softirq+0x2d/0x30
  [do_IRQ+30/48] do_IRQ+0x1e/0x30
  [common_interrupt+26/32] common_interrupt+0x1a/0x20
  [default_idle+0/64] default_idle+0x0/0x40
  [default_idle+44/64] default_idle+0x2c/0x40
  [cpu_idle+51/64] cpu_idle+0x33/0x40
  [start_kernel+331/368] start_kernel+0x14b/0x170
  [unknown_bootoption+0/432] unknown_bootoption+0x0/0x1b0
 DMA per-cpu:
 cpu 0 hot: low 2, high 6, batch 1
 cpu 0 cold: low 0, high 2, batch 1
 cpu 1 hot: low 2, high 6, batch 1
 cpu 1 cold: low 0, high 2, batch 1
 Normal per-cpu:
 cpu 0 hot: low 32, high 96, batch 16
 cpu 0 cold: low 0, high 32, batch 16
 cpu 1 hot: low 32, high 96, batch 16
 cpu 1 cold: low 0, high 32, batch 16
 HighMem per-cpu:
 cpu 0 hot: low 32, high 96, batch 16
 cpu 0 cold: low 0, high 32, batch 16
 cpu 1 hot: low 32, high 96, batch 16
 cpu 1 cold: low 0, high 32, batch 16
 
 Free pages:        4616kB (1600kB HighMem)
 Active:504159 inactive:454759 dirty:20020 writeback:115 unstable:0 free:1154 slab:50758 mapped:489095 pagetables:1222
 DMA free:56kB min:144kB low:288kB high:432kB active:1936kB inactive:4932kB present:16384kB pages_scanned:32 all_unreclaimable? no
 protections[]: 0 0 0
 Normal free:2960kB min:8044kB low:16088kB high:24132kB active:492320kB inactive:166992kB present:901120kB pages_scanned:62 all_unreclaimable? no
 protections[]: 0 0 0
 HighMem free:1600kB min:512kB low:1024kB high:1536kB active:1522380kB inactive:1647112kB present:3178432kB pages_scanned:0 all_unreclaimable? no
 protections[]: 0 0 0
 DMA: 0*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56kB
 Normal: 0*4kB 0*8kB 1*16kB 0*32kB 2*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2960kB
 HighMem: 6*4kB 3*8kB 41*16kB 0*32kB 6*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1600kB
 Swap cache: add 154147, delete 151810, find 29532/39794, race 0+0

Well you've definitely used up all the memory which is available for atomic
allocations.  Are you using an increased /proc/sys/vm/min_free_kbytes there?

As for the application collapse: dunno.  Maybe networking broke.  It would
be interesting to test Linus's current tree, at
ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.10-rc1-bk19.gz

On Tue, Nov 09, 2004 at 05:39:20PM -0800, Andrew Morton wrote:

Well you've definitely used up all the memory which is available for atomic
allocations.  Are you using an increased /proc/sys/vm/min_free_kbytes there?

Yes, vm.min_free_kbytes=8192.
For other vm-settings find sysctl.conf attached.

Netdev: tg3 BCM5704r03, TSO off, ~32kpps rx, ~35kpps tx, ~2 rx errors/s

As for the application collapse: dunno.  Maybe networking broke.  It would
be interesting to test Linus's current tree, at
ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.10-rc1-bk19.gz

Will try that tomorrow. Would you suggest printing out show_free_areas();
there too? I don't know what kind of an overhead that will generate on
subsequent stack traces.

	Stefan

Stefan Schmidt [off-list ref] wrote:

quoted

As for the application collapse: dunno.  Maybe networking broke.  It would

 > be interesting to test Linus's current tree, at
 > ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.10-rc1-bk19.gz
 Will try that tomorrow. Would you suggest printing out show_free_areas();
 there too? I don't know what kind of an overhead that will generate on
 subsequent stack traces.

I don't think it'd help much - we know what's happening.

It would be interesting to keep increasing min_free_kbytes, see if you can
characterise the system's response to this setting.

Stefan Schmidt wrote:

On Tue, Nov 09, 2004 at 05:39:20PM -0800, Andrew Morton wrote:

quoted

Well you've definitely used up all the memory which is available for atomic
allocations.  Are you using an increased /proc/sys/vm/min_free_kbytes there?

Yes, vm.min_free_kbytes=8192.
For other vm-settings find sysctl.conf attached.

Netdev: tg3 BCM5704r03, TSO off, ~32kpps rx, ~35kpps tx, ~2 rx errors/s

quoted

As for the application collapse: dunno.  Maybe networking broke.  It would
be interesting to test Linus's current tree, at
ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.10-rc1-bk19.gz

Will try that tomorrow. Would you suggest printing out show_free_areas();
there too? I don't know what kind of an overhead that will generate on
subsequent stack traces.

Stefan,
Can you try the following patch, please? It is diffed against 2.6.10-rc1,
but I think it should apply to -mm kernels as well.

Basically 2.6.8 and earlier kernels had some quirks in the page allocator
that would allow for example, a large portion of "DMA" memory to be reserved
for network memory allocations (atomic allocations). After 'fixing' this
problem, 2.6.9 is effectively left with about a quarter the amount of memory
reserved for network allocations compared with 2.6.8.

The following patch roughly restores parity there. Thanks.

Nick

On Wed, Nov 10, 2004 at 03:24:10PM +1100, Nick Piggin wrote:

Can you try the following patch, please? It is diffed against 2.6.10-rc1,
but I think it should apply to -mm kernels as well.

Basically 2.6.8 and earlier kernels had some quirks in the page allocator
that would allow for example, a large portion of "DMA" memory to be reserved
for network memory allocations (atomic allocations). After 'fixing' this
problem, 2.6.9 is effectively left with about a quarter the amount of memory
reserved for network allocations compared with 2.6.8.

The following patch roughly restores parity there. Thanks.

I applied the patch to 2.6.10-rc1-mm4 and the application froze again, but i
just remembered that i changed a kernel-option in mm4 and forgot about that
yesterday:
I unset CONFIG_PACKET_MMAP and i suppose this could have this kind of effect
on high connection rates.
I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze
for some hours at this load we can blame at least this issue (-1 EAGAIN) on
that parameter.

My variation of Harrisberger's Fourth Law of the Lab:
 Experience is directly proportional to the amount of braincells ruined.

*ouch*,
	Stefan

On Wed, Nov 10, 2004 at 11:28:54AM +0100, Stefan Schmidt wrote:

quoted

Can you try the following patch, please? It is diffed against 2.6.10-rc1,
but I think it should apply to -mm kernels as well.

I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will try
latest bk next.

I unset CONFIG_PACKET_MMAP and i suppose this could have this kind of effect
on high connection rates.
I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze
for some hours at this load we can blame at least this issue (-1 EAGAIN) on
that parameter.

Nope, that didn't change anything, still getting EAGAIN, checked two times.

	Stefan

On Wed, Nov 10, 2004 at 01:06:24PM +0100, Stefan Schmidt wrote:

On Wed, Nov 10, 2004 at 11:28:54AM +0100, Stefan Schmidt wrote:

quoted

quoted

Can you try the following patch, please? It is diffed against 2.6.10-rc1,
but I think it should apply to -mm kernels as well.

I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will try
latest bk next.

quoted

I unset CONFIG_PACKET_MMAP and i suppose this could have this kind of effect
on high connection rates.
I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze
for some hours at this load we can blame at least this issue (-1 EAGAIN) on
that parameter.

Nope, that didn't change anything, still getting EAGAIN, checked two times.

Hi Stefan,

Its not clear to me - do you have Nick's watermark patch in? 

On Wed, Nov 10, 2004 at 06:58:31AM -0200, Marcelo Tosatti wrote:

quoted

quoted

quoted

Can you try the following patch, please? It is diffed against 2.6.10-rc1,

I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will try
latest bk next.

quoted

quoted

I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze
for some hours at this load we can blame at least this issue (-1 EAGAIN) on
that parameter.

Nope, that didn't change anything, still getting EAGAIN, checked two times.

Its not clear to me - do you have Nick's watermark patch in? 

Yes i have vm.min_free_kbytes=8192 and Nick's patch in mm4. I'll try
rc1-bk19 with his restore-atomic-buffer patch in a few minutes.

	Stefan
-- 
The reason computer chips are so small is computers don't eat much.

On Wed, Nov 10, 2004 at 01:48:11PM +0100, Stefan Schmidt wrote:

On Wed, Nov 10, 2004 at 06:58:31AM -0200, Marcelo Tosatti wrote:

quoted

quoted

quoted

quoted

Can you try the following patch, please? It is diffed against 2.6.10-rc1,

I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will try
latest bk next.

quoted

quoted

quoted

I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze
for some hours at this load we can blame at least this issue (-1 EAGAIN) on
that parameter.

Nope, that didn't change anything, still getting EAGAIN, checked two times.

Its not clear to me - do you have Nick's watermark patch in? 

Yes i have vm.min_free_kbytes=8192 and Nick's patch in mm4. I'll try
rc1-bk19 with his restore-atomic-buffer patch in a few minutes.

Stefan, 

Please always run your tests with show_free_area() call at the 
page allocation failure path.

I fully disagree with Andrew when he says  
"I don't think it'd help much - we know what's happening."

Stefan Schmidt wrote:

On Wed, Nov 10, 2004 at 06:58:31AM -0200, Marcelo Tosatti wrote:

quoted

quoted

quoted

quoted

Can you try the following patch, please? It is diffed against 2.6.10-rc1,

I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will try
latest bk next.

quoted

quoted

quoted

I set it back to CONFIG_PACKET_MMAP=y and if the application does not freeze
for some hours at this load we can blame at least this issue (-1 EAGAIN) on
that parameter.

Nope, that didn't change anything, still getting EAGAIN, checked two times.

Its not clear to me - do you have Nick's watermark patch in? 

Yes i have vm.min_free_kbytes=8192 and Nick's patch in mm4. I'll try
rc1-bk19 with his restore-atomic-buffer patch in a few minutes.

You'll actually want to increase min_free_kbytes in order to have the same
amount of memory free as 2.6.8 did.

Start by applying my patch and using the default min_free_kbytes. Then 
increase
it until the page allocation failures stop, and let us know what the end 
result
was.

BTW we should probably have a message in the page allocation failure path
to tell people to try increasing /proc/sys/vm/min_free_kbytes...

On Thu, Nov 11, 2004 at 12:23:13PM +1100, Nick Piggin wrote:

Stefan Schmidt wrote:

quoted

On Wed, Nov 10, 2004 at 06:58:31AM -0200, Marcelo Tosatti wrote:

quoted

quoted

quoted

quoted

Can you try the following patch, please? It is diffed against 
2.6.10-rc1,

I did. No apparent change with mm4 and vm.min_free_kbytes = 8192. I will 
try
latest bk next.

quoted

quoted

quoted

I set it back to CONFIG_PACKET_MMAP=y and if the application does not 
freeze
for some hours at this load we can blame at least this issue (-1 
EAGAIN) on
that parameter.

Nope, that didn't change anything, still getting EAGAIN, checked two 
times.

Its not clear to me - do you have Nick's watermark patch in? 

Yes i have vm.min_free_kbytes=8192 and Nick's patch in mm4. I'll try
rc1-bk19 with his restore-atomic-buffer patch in a few minutes.

You'll actually want to increase min_free_kbytes in order to have the same
amount of memory free as 2.6.8 did.

Start by applying my patch and using the default min_free_kbytes. Then 
increase
it until the page allocation failures stop, and let us know what the end 
result
was.

BTW we should probably have a message in the page allocation failure path
to tell people to try increasing /proc/sys/vm/min_free_kbytes...

We are having a similar problem (at least i think).
although I have up'd the min_free_kbytes to what was suggested in this
post and havent seen these messages _yet_. 


Nov  7 13:02:23 aragorn kernel: swapper: page allocation failure.
order:1, mode:0x20
Nov  7 13:02:23 aragorn kernel: [<c01356f2>] __alloc_pages+0x1b3/0x358
Nov  7 13:02:23 aragorn kernel: [<c01358bc>] __get_free_pages+0x25/0x3f
Nov  7 13:02:23 aragorn kernel: [<c01389b0>] kmem_getpages+0x21/0xc9
Nov  7 13:02:23 aragorn kernel: [<c013968f>] cache_grow+0xab/0x14d
Nov  7 13:02:23 aragorn kernel: [<c01398a5>]
cache_alloc_refill+0x174/0x219
Nov  7 13:02:23 aragorn kernel: [<c0139b17>] kmem_cache_alloc+0x4b/0x4d
Nov  7 13:02:23 aragorn kernel: [<c0238118>] sk_alloc+0x34/0xa7
Nov  7 13:02:23 aragorn kernel: [<c026d485>]
tcp_create_openreq_child+0x34/0x558
Nov  7 13:02:23 aragorn kernel: [<c0269d81>]
tcp_v4_syn_recv_sock+0x47/0x2ea
Nov  7 13:02:23 aragorn kernel: [<c026db74>] tcp_check_req+0x1cb/0x4df
Nov  7 13:02:23 aragorn kernel: [<f88b64f0>] tg3_start_xmit+0x3f4/0x4f5
[tg3]
Nov  7 13:02:23 aragorn kernel: [<f88b64f0>] tg3_start_xmit+0x3f4/0x4f5
[tg3]
Nov  7 13:02:23 aragorn kernel: [<c023ec88>] dev_queue_xmit+0x120/0x284
Nov  7 13:02:23 aragorn kernel: [<c0248a0e>] qdisc_restart+0x17/0x1d9
Nov  7 13:02:23 aragorn kernel: [<c023ec88>] dev_queue_xmit+0x120/0x284
Nov  7 13:02:23 aragorn kernel: [<c0252a95>] ip_finish_output+0xbb/0x1b9
Nov  7 13:02:23 aragorn kernel: [<c025310d>] ip_queue_xmit+0x3f1/0x500
Nov  7 13:02:23 aragorn kernel: [<c01122a9>] try_to_wake_up+0x1de/0x26c
Nov  7 13:02:23 aragorn kernel: [<c0111c5c>] recalc_task_prio+0x93/0x188
Nov  7 13:02:23 aragorn kernel: [<c0111de1>] activate_task+0x90/0xa4
Nov  7 13:02:23 aragorn kernel: [<c01122a9>] try_to_wake_up+0x1de/0x26c
Nov  7 13:02:23 aragorn kernel: [<c0113c3d>] __wake_up_common+0x3f/0x5e
Nov  7 13:02:23 aragorn kernel: [<c0113c9c>] __wake_up+0x40/0x56
Nov  7 13:02:23 aragorn kernel: [<c026a08a>] tcp_v4_hnd_req+0x66/0x20a
Nov  7 13:02:23 aragorn kernel: [<c026dedb>] tcp_child_process+0x53/0xc4
Nov  7 13:02:23 aragorn kernel: [<c026a45e>] tcp_v4_do_rcv+0xd7/0x12d
Nov  7 13:02:23 aragorn kernel: [<c026ab8c>] tcp_v4_rcv+0x6d8/0x90f
Nov  7 13:02:23 aragorn kernel: [<c0111c5c>] recalc_task_prio+0x93/0x188
Nov  7 13:02:23 aragorn kernel: [<c024fd92>] ip_local_deliver+0xb5/0x201
Nov  7 13:02:23 aragorn kernel: [<c025021a>] ip_rcv+0x33c/0x47e
Nov  7 13:02:23 aragorn kernel: [<c0112fe8>]
find_busiest_group+0xe4/0x316
Nov  7 13:02:23 aragorn kernel: [<c023f2db>]
netif_receive_skb+0x1ac/0x242
Nov  7 13:02:23 aragorn kernel: [<c0239309>] alloc_skb+0x47/0xe0
Nov  7 13:02:23 aragorn kernel: [<f88b5a66>] tg3_rx+0x2a7/0x3fa [tg3]
Nov  7 13:02:23 aragorn kernel: [<f88b5c46>] tg3_poll+0x8d/0x130 [tg3]
Nov  7 13:02:23 aragorn kernel: [<c023f4f3>] net_rx_action+0x77/0xf6
Nov  7 13:02:23 aragorn kernel: [<c011b9cf>] __do_softirq+0xb7/0xc6
Nov  7 13:02:23 aragorn kernel: [<c011ba0b>] do_softirq+0x2d/0x2f
Nov  7 13:02:23 aragorn kernel: [<c0106adf>] do_IRQ+0x112/0x130
Nov  7 13:02:23 aragorn kernel: [<c010494c>] common_interrupt+0x18/0x20
Nov  7 13:02:23 aragorn kernel: [<c0101f1e>] default_idle+0x0/0x2c
Nov  7 13:02:23 aragorn kernel: [<c0101f47>] default_idle+0x29/0x2c
Nov  7 13:02:23 aragorn kernel: [<c0101fbc>] cpu_idle+0x3f/0x58
Nov  7 13:02:23 aragorn kernel: [<c031e9d5>] start_kernel+0x16e/0x189
Nov  7 13:02:23 aragorn kernel: [<c031e49f>]
unknown_bootoption+0x0/0x15c


Nov  7 13:02:23 aragorn kernel: swapper: page allocation failure.
order:1, mode:0x20
Nov  7 13:02:23 aragorn kernel: [<c01356f2>] __alloc_pages+0x1b3/0x358
Nov  7 13:02:23 aragorn kernel: [<c01358bc>] __get_free_pages+0x25/0x3f
Nov  7 13:02:23 aragorn kernel: [<c01389b0>] kmem_getpages+0x21/0xc9
Nov  7 13:02:23 aragorn syslog-ng[1574]: Error accepting AF_UNIX
connection, opened connections: 100, max: 100
Nov  7 13:02:23 aragorn kernel: [<c013968f>] cache_grow+0xab/0x14d
Nov  7 13:02:23 aragorn kernel: [<c01398a5>]
cache_alloc_refill+0x174/0x219
Nov  7 13:02:23 aragorn kernel: [<c0139b17>] kmem_cache_alloc+0x4b/0x4d
Nov  7 13:02:23 aragorn kernel: [<c0238118>] sk_alloc+0x34/0xa7
Nov  7 13:02:23 aragorn kernel: [<c026d485>]
tcp_create_openreq_child+0x34/0x558
Nov  7 13:02:23 aragorn kernel: [<c0269d81>]
tcp_v4_syn_recv_sock+0x47/0x2ea
Nov  7 13:02:23 aragorn kernel: [<c026db74>] tcp_check_req+0x1cb/0x4df
Nov  7 13:02:23 aragorn kernel: [<f88b64f0>] tg3_start_xmit+0x3f4/0x4f5
[tg3]
Nov  7 13:02:23 aragorn kernel: [<f88b64f0>] tg3_start_xmit+0x3f4/0x4f5
[tg3]
Nov  7 13:02:23 aragorn kernel: [<c023ec88>] dev_queue_xmit+0x120/0x284
Nov  7 13:02:23 aragorn kernel: [<c0248a0e>] qdisc_restart+0x17/0x1d9
Nov  7 13:02:23 aragorn kernel: [<c023ec88>] dev_queue_xmit+0x120/0x284
Nov  7 13:02:23 aragorn kernel: [<c0252a95>] ip_finish_output+0xbb/0x1b9
Nov  7 13:02:23 aragorn kernel: [<c0111c5c>] recalc_task_prio+0x93/0x188
Nov  7 13:02:23 aragorn kernel: [<c0111c5c>] recalc_task_prio+0x93/0x188
Nov  7 13:02:23 aragorn kernel: [<f88b64f0>] tg3_start_xmit+0x3f4/0x4f5
[tg3]
Nov  7 13:02:23 aragorn kernel: [<c027dc75>]
fib_validate_source+0x22c/0x257
Nov  7 13:02:23 aragorn kernel: [<c0248a0e>] qdisc_restart+0x17/0x1d9
Nov  7 13:02:23 aragorn kernel: [<c023ec88>] dev_queue_xmit+0x120/0x284
Nov  7 13:02:23 aragorn kernel: [<c0252a95>] ip_finish_output+0xbb/0x1b9
Nov  7 13:02:23 aragorn kernel: [<c025310d>] ip_queue_xmit+0x3f1/0x500
Nov  7 13:02:23 aragorn kernel: [<c026a08a>] tcp_v4_hnd_req+0x66/0x20a
Nov  7 13:02:23 aragorn kernel: [<c024ea28>]
ip_route_output_flow+0x2f/0x9d
Nov  7 13:02:23 aragorn kernel: [<c026a45e>] tcp_v4_do_rcv+0xd7/0x12d
Nov  7 13:02:23 aragorn kernel: [<c026ab8c>] tcp_v4_rcv+0x6d8/0x90f
Nov  7 13:02:23 aragorn kernel: [<c024fd92>] ip_local_deliver+0xb5/0x201
Nov  7 13:02:23 aragorn kernel: [<c025021a>] ip_rcv+0x33c/0x47e
Nov  7 13:02:23 aragorn kernel: [<c023f2db>]
netif_receive_skb+0x1ac/0x242
Nov  7 13:02:23 aragorn kernel: [<c0239309>] alloc_skb+0x47/0xe0
Nov  7 13:02:23 aragorn kernel: [<f88b5a66>] tg3_rx+0x2a7/0x3fa [tg3]
Nov  7 13:02:23 aragorn kernel: [<f88b5c46>] tg3_poll+0x8d/0x130 [tg3]
Nov  7 13:02:23 aragorn kernel: [<c023f4f3>] net_rx_action+0x77/0xf6
Nov  7 13:02:23 aragorn kernel: [<c011b9cf>] __do_softirq+0xb7/0xc6
Nov  7 13:02:23 aragorn kernel: [<c0106adf>] do_IRQ+0x112/0x130
Nov  7 13:02:23 aragorn kernel: [<c010494c>] common_interrupt+0x18/0x20
Nov  7 13:02:23 aragorn kernel: [<c0101f1e>] default_idle+0x0/0x2c
Nov  7 13:02:23 aragorn kernel: [<c0101f47>] default_idle+0x29/0x2c
Nov  7 13:02:23 aragorn kernel: [<c0101fbc>] cpu_idle+0x3f/0x58
Nov  7 13:02:23 aragorn kernel: [<c031e9d5>] start_kernel+0x16e/0x189
Nov  7 13:02:23 aragorn kernel: [<c031e49f>]
unknown_bootoption+0x0/0x15c

$ lspci
0000:00:00.0 Host bridge: ServerWorks CMIC-LE Host Bridge (GC-LE
chipset) (rev 33)
0000:00:00.1 Host bridge: ServerWorks CMIC-LE Host Bridge (GC-LE
chipset)
0000:00:00.2 Host bridge: ServerWorks CMIC-LE Host Bridge (GC-LE
chipset)
0000:00:0e.0 VGA compatible controller: ATI Technologies Inc Rage XL
(rev 27)
0000:00:0f.0 Host bridge: ServerWorks CSB5 South Bridge (rev 93)
0000:00:0f.1 IDE interface: ServerWorks CSB5 IDE Controller (rev 93)
0000:00:0f.2 USB Controller: ServerWorks OSB4/CSB5 OHCI USB Controller
(rev 05)
0000:00:0f.3 ISA bridge: ServerWorks CSB5 LPC bridge
0000:00:10.0 Host bridge: ServerWorks CIOB-E I/O Bridge with Gigabit
Ethernet (rev 12)
0000:00:10.2 Host bridge: ServerWorks CIOB-E I/O Bridge with Gigabit
Ethernet (rev 12)
0000:00:11.0 Host bridge: ServerWorks CIOB-X2 PCI-X I/O Bridge (rev 05)
0000:00:11.2 Host bridge: ServerWorks CIOB-X2 PCI-X I/O Bridge (rev 05)
0000:02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
Gigabit Ethernet (rev 02)
0000:02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
Gigabit Ethernet (rev 02)
0000:04:03.0 RAID bus controller: Dell PowerEdge Expandable RAID
controller 4/Di (rev 02)