I'm willing to guess at this point that this atomic failure is not a bug, the 
only bug is that we print the warning message, potentially slowing things 
down.  I'd like to see a correct backtrace first.

Do you detect any slowdown in your system when you're getting these messages? 
I wouldn't expect so from what you've described so far.

I don't notice any slowdown in the system, but certain operations hang. 
  Such as vmstat, ps, and our SMTP server.  They lock up completely. 
The load average is at 20, but the CPU is completely idle.
Here's another attempt at ksymoops output:

ksymoops 2.4.0 on i686 2.4.9.  Options used
      -V (default)
      -k /proc/ksyms (default)
      -l /proc/modules (default)
      -o /lib/modules/2.4.9/ (default)
      -m /boot/System.map-2.4.9 (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Aug 29 19:00:17 dhcp80-252 kernel: Call Trace: [<c0129b60>] [<c0129e0e>] 
[<c0126e7a>] [<c0127029>] [<c01f8242>]
Aug 29 19:00:17 dhcp80-252 kernel:    [<c021c02a>] [<c0219a6b>] 
[<c021c6b1>] [<c01a9024>] [<c0206464>] [<c0200413>]
Aug 29 19:00:17 dhcp80-252 kernel:    [<c0206464>] [<c01fc517>] 
[<c0206464>] [<c020651d>] [<c0206464>] [<c0206935>]
Aug 29 19:00:17 dhcp80-252 kernel:    [<c0106bf4>] [<c0206400>] 
[<c01fa441>] [<c0206935>] [<c01fa441>] [<c01f8fc3>]
Aug 29 19:00:17 dhcp80-252 kernel:    [<c01f8fd7>] [<c01f910b>] 
[<c0219c9e>] [<c0219f3d>] [<c021a35b>] [<c020414f>]
Aug 29 19:00:17 dhcp80-252 kernel:    [<c02044a9>] [<c01a8de4>] 
[<c01fca3d>] [<c0116f6d>] [<c0107ff4>] [<c01051a0>]
Aug 29 19:00:17 dhcp80-252 kernel:    [<c01051a0>] [<c0106bf4>] 
[<c01051a0>] [<c01051a0>] [<c01051c3>] [<c0105224>]
Aug 29 19:00:18 dhcp80-252 kernel:    [<c0105000>] [<c0105027>]
Aug 29 19:00:18 dhcp80-252 kernel: Call Trace: [<c0129b60>] [<c0129e0e>] 
[<c0126e7a>] [<c0127029>] [<c01f8242>]
Aug 29 19:00:18 dhcp80-252 kernel:    [<c021c02a>] [<c0219a6b>] 
[<c021c6b1>] [<c01a9024>] [<c01a954e>] [<c0206464>]
Aug 29 19:00:18 dhcp80-252 kernel:    [<c0200413>] [<c0206464>] 
[<c01fc517>] [<c0206464>] [<c020651d>] [<c0206464>]
Aug 29 19:00:18 dhcp80-252 kernel:    [<c0206935>] [<c01fc517>] 
[<c0206464>] [<c020651d>] [<c0206464>] [<c0214500>]
Aug 29 19:00:18 dhcp80-252 kernel:    [<c0214627>] [<c02150bf>] 
[<c01f8fc3>] [<c01f8fd7>] [<c01f910b>] [<c0219c9e>]
Aug 29 19:00:18 dhcp80-252 kernel:    [<c0219f3d>] [<c021a35b>] 
[<c020414f>] [<c02044a9>] [<c01a8de4>] [<c01fca3d>]
Aug 29 19:00:18 dhcp80-252 kernel:    [<c0116f6d>] [<c0107ff4>] 
[<c01051a0>] [<c01051a0>] [<c0106bf4>] [<c01051a0>]
Aug 29 19:00:18 dhcp80-252 kernel:    [<c01051a0>] [<c01051c3>] 
[<c0105224>] [<c0105000>] [<c0105027>]
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; c0129b60 <_alloc_pages+18/1c>
Trace; c0129e0e <__get_free_pages+a/18>
Trace; c0126e7a <kmem_cache_grow+ce/234>
Trace; c0127029 <kmem_cache_alloc+49/58>
Trace; c01f8242 <sk_alloc+12/58>
Trace; c021c02a <tcp_create_openreq_child+16/44c>
Trace; c0219a6b <tcp_v4_syn_recv_sock+57/248>
Trace; c021c6b1 <tcp_check_req+251/380>
Trace; c01a9024 <speedo_refill_rx_buf+40/20c>
Trace; c0206464 <ip_output+0/f0>
Trace; c0200413 <qdisc_restart+13/c8>
Trace; c0206464 <ip_output+0/f0>
Trace; c01fc517 <dev_queue_xmit+117/264>
Trace; c0206464 <ip_output+0/f0>
Trace; c020651d <ip_output+b9/f0>
Trace; c0206464 <ip_output+0/f0>
Trace; c0206935 <ip_queue_xmit+3e1/540>
Trace; c0106bf4 <ret_from_intr+0/7>
Trace; c0206400 <ip_mc_output+108/16c>
Trace; c01fa441 <skb_copy_and_csum_bits+51/35c>
Trace; c0206935 <ip_queue_xmit+3e1/540>
Trace; c01fa441 <skb_copy_and_csum_bits+51/35c>
Trace; c01f8fc3 <skb_release_data+67/70>
Trace; c01f8fd7 <kfree_skbmem+b/58>
Trace; c01f910b <__kfree_skb+e7/f0>
Trace; c0219c9e <tcp_v4_hnd_req+42/150>
Trace; c0219f3d <tcp_v4_do_rcv+91/108>
Trace; c021a35b <tcp_v4_rcv+3a7/618>
Trace; c020414f <ip_local_deliver+eb/164>
Trace; c02044a9 <ip_rcv+2e1/338>
Trace; c01a8de4 <speedo_interrupt+b0/2b0>
Trace; c01fca3d <net_rx_action+135/208>
Trace; c0116f6d <do_softirq+5d/ac>
Trace; c0107ff4 <do_IRQ+98/a8>
Trace; c01051a0 <default_idle+0/28>
Trace; c01051a0 <default_idle+0/28>
Trace; c0106bf4 <ret_from_intr+0/7>
Trace; c01051a0 <default_idle+0/28>
Trace; c01051a0 <default_idle+0/28>
Trace; c01051c3 <default_idle+23/28>
Trace; c0105224 <cpu_idle+3c/50>
Trace; c0105000 <_stext+0/0>
Trace; c0105027 <rest_init+27/28>
Trace; c0129b60 <_alloc_pages+18/1c>
Trace; c0129e0e <__get_free_pages+a/18>
Trace; c0126e7a <kmem_cache_grow+ce/234>
Trace; c0127029 <kmem_cache_alloc+49/58>
Trace; c01f8242 <sk_alloc+12/58>
Trace; c021c02a <tcp_create_openreq_child+16/44c>
Trace; c0219a6b <tcp_v4_syn_recv_sock+57/248>
Trace; c021c6b1 <tcp_check_req+251/380>
Trace; c01a9024 <speedo_refill_rx_buf+40/20c>
Trace; c01a954e <speedo_rx+326/344>
Trace; c0206464 <ip_output+0/f0>
Trace; c0200413 <qdisc_restart+13/c8>
Trace; c0206464 <ip_output+0/f0>
Trace; c01fc517 <dev_queue_xmit+117/264>
Trace; c0206464 <ip_output+0/f0>
Trace; c020651d <ip_output+b9/f0>
Trace; c0206464 <ip_output+0/f0>
Trace; c0206935 <ip_queue_xmit+3e1/540>
Trace; c01fc517 <dev_queue_xmit+117/264>
Trace; c0206464 <ip_output+0/f0>
Trace; c020651d <ip_output+b9/f0>
Trace; c0206464 <ip_output+0/f0>
Trace; c0214500 <tcp_transmit_skb+36c/540>
Trace; c0214627 <tcp_transmit_skb+493/540>
Trace; c02150bf <tcp_write_xmit+18f/2dc>
Trace; c01f8fc3 <skb_release_data+67/70>
Trace; c01f8fd7 <kfree_skbmem+b/58>
Trace; c01f910b <__kfree_skb+e7/f0>
Trace; c0219c9e <tcp_v4_hnd_req+42/150>
Trace; c0219f3d <tcp_v4_do_rcv+91/108>
Trace; c021a35b <tcp_v4_rcv+3a7/618>
Trace; c020414f <ip_local_deliver+eb/164>
Trace; c02044a9 <ip_rcv+2e1/338>
Trace; c01a8de4 <speedo_interrupt+b0/2b0>
Trace; c01fca3d <net_rx_action+135/208>
Trace; c0116f6d <do_softirq+5d/ac>
Trace; c0107ff4 <do_IRQ+98/a8>
Trace; c01051a0 <default_idle+0/28>
Trace; c01051a0 <default_idle+0/28>
Trace; c0106bf4 <ret_from_intr+0/7>
Trace; c01051a0 <default_idle+0/28>
Trace; c01051a0 <default_idle+0/28>
Trace; c01051c3 <default_idle+23/28>
Trace; c0105224 <cpu_idle+3c/50>
Trace; c0105000 <_stext+0/0>
Trace; c0105027 <rest_init+27/28>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/