Re: Kernel crash in 2.6.0-test9-mm3

Could this be happening on an SMP system only ? If so, e100intr routine
services tx queues (e100_tx_srv)
without holding a lock. Can't multiple rx interrupts be scheduled on
different cpus at the same time, and
each execute dev_kfree_skb_irq() which decrements the ref count too many
times ? But the softirq handler
(net_tx_action) seems to clean up the skb once as the dec_test returns 1
only if count is zero, so I don't see
where the dst ref is being decremented wrongly in this case.

Can someone explain why the intr handler doesn't need locks to stop other
intr on different cpu's from going
through the same devices memory at the same time ?

Thanks,

- KK



|---------+---------------------------->
|         |           Reuben Farrelly  |
|         |           <reuben-linux@reu|
|         |           b.net>           |
|         |           Sent by:         |
|         |           netdev-bounce@oss|
|         |           .sgi.com         |
|         |                            |
|         |                            |
|         |           11/18/2003 05:22 |
|         |           PM               |
|         |                            |
|---------+---------------------------->
  >-----------------------------------------------------------------------------------------------------------------|
  |                                                                                                                 |
  |       To:       "David S. Miller" [off-list ref], Andrew Morton [off-list ref]                             |
  |       cc:       netdev@oss.sgi.com                                                                              |
  |       Subject:  Re: Kernel crash in 2.6.0-test9-mm3                                                             |
  |                                                                                                                 |
  >-----------------------------------------------------------------------------------------------------------------|




FWIW I'm compiling with:

[root@tornado log]# gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.3.2/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--disable-checking --with-system-zlib --enable-__cxa_atexit
--host=i386-redhat-linux
Thread model: posix
gcc version 3.3.2 20031107 (Red Hat Linux 3.3.2-2)
[root@tornado log]#

Reuben


At 13:49 19/11/2003, David S. Miller wrote:

On Tue, 18 Nov 2003 11:01:39 -0800
Andrew Morton [off-list ref] wrote:

quoted

It's one for the networking guys.

The mm kernels have a patch which detects when atomic_dec_and_test
takes an atomic_t negative - it is assumed that this is a bug so
a warning is generated.

Andrew I've analyzed this a bit.  This is incredible evidence in
these dumps that either there is a bug in Linus's atomic_dec_and_test()
debugging hack or GCC is miscompiling it in certain cases with certain
versions of the compiler.

Look at this:

quoted

quoted

Nov 18 23:09:00 tornado kernel:  [<c029203c>]

skb_release_data+0x14c/0x160

quoted

quoted

Nov 18 23:09:00 tornado kernel:  [<c0292063>] kfree_skbmem+0x13/0x30
Nov 18 23:09:00 tornado kernel:  [<c0292138>] __kfree_skb+0xb8/0x1b0
Nov 18 23:09:00 tornado kernel:  [<c0218815>] e100intr+0x1e5/0x290

Ok, releasing an SKB data area twice.

quoted

quoted

Nov 18 23:09:00 tornado kernel: BUG: dst underflow 0: c02921ef

Freeing a 'dst' entry one too many times.

quoted

quoted

Nov 18 23:09:00 tornado kernel: Attempt to release alive inet socket

dfd4c780

A socket refcount dropping to zero too early, before it's marked dead.

These last two problems are very serious errors, and would have
printed out debugging messages before the atomic_dec_and_test() patch.
If these last two messages don't show up without the
atomic_dec_and_test() debugging patch applied, well there you
go... :-)

In that debugging patch, I'm wondering something about x86.
When one goes "sete %reg; sets %reg" does the first 'sete' modify
the condition codes by chance?  Probably not...

On Tue, 18 Nov 2003 18:22:42 -0800
Krishna Kumar [off-list ref] wrote:

Could this be happening on an SMP system only ? If so, e100intr routine
services tx queues (e100_tx_srv) without holding a lock.

[ Scott we have a bug report, and we're trying to determine if the
  cause is that the e100 driver frees a TX SKB multiple times due
  to some race or other problem in current 2.6.x ]

That's a very good point, and I looked a bit in this area.

When e100intr() is doing it's work, it disables and clears
the interrupt, only after doing RX and TX processing does
it reenable chip interrupts via e100_set_intr_mask().

That is my analysis of the situation.

However, with things like IOAPIC and such, it might be possible
for two cpus to enter e100intr() simultaneously, both read
the same status, both see that the interrupt is pending, and
both thus process the interrupt and race with each other.

Scott, what prevents the above from happening?

The system is a UP system..

# CONFIG_SMP is not set
CONFIG_PREEMPT=y
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y

I've put the full .config up at http://www.reub.net/linux/

Reuben



At 15:22 19/11/2003, Krishna Kumar wrote:

Could this be happening on an SMP system only ? If so, e100intr routine
services tx queues (e100_tx_srv)
without holding a lock. Can't multiple rx interrupts be scheduled on
different cpus at the same time, and
each execute dev_kfree_skb_irq() which decrements the ref count too many
times ? But the softirq handler
(net_tx_action) seems to clean up the skb once as the dec_test returns 1
only if count is zero, so I don't see
where the dst ref is being decremented wrongly in this case.