Re: [Bug #14378] Problems with net/core/skbuff.c

David Miller ha scritto:

From: "Rafael J. Wysocki" <redacted>
Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)

  

quoted

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=14378
Subject		: Problems with net/core/skbuff.c
Submitter	: Massimo Cetra [off-list ref]
Date		: 2009-10-08 14:51 (4 days old)
References	: http://marc.info/?l=linux-kernel&m=125501488220358&w=4
    

I don't know what to do about this one.

The user indicates that they have the vserver patches applied,
so maybe there is some interaction with that stuff.
  

Actually i found another oops which is very similar to the previous one.
Here, vserver is not involved, and the problem starts at drbd which 
lives in kernel space (the other oops started at ocfs2).

Both ocfs2 and drbd make heavy use of network I/O so i guess the problem 
is something in the network layer.

Anything i can do to help to debugging and solving this issue ?

Thanks
Max

Massimo Cetra ha scritto:

David Miller ha scritto:

quoted

From: "Rafael J. Wysocki" <redacted>
Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)

 

quoted

Bug-Entry    : http://bugzilla.kernel.org/show_bug.cgi?id=14378
Subject        : Problems with net/core/skbuff.c
Submitter    : Massimo Cetra [off-list ref]
Date        : 2009-10-08 14:51 (4 days old)
References    : http://marc.info/?l=linux-kernel&m=125501488220358&w=4
    

I don't know what to do about this one.

The user indicates that they have the vserver patches applied,
so maybe there is some interaction with that stuff.
  

Actually i found another oops which is very similar to the previous one.

And here it is another one, this time triggered by postfix, where mor 
drbd nor vserver are involved.
This is not the same server where the other oopses were grabbed.

Max

Massimo Cetra a écrit :

David Miller ha scritto:

quoted

From: "Rafael J. Wysocki" <redacted>
Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST)

 

quoted

Bug-Entry    : http://bugzilla.kernel.org/show_bug.cgi?id=14378
Subject        : Problems with net/core/skbuff.c
Submitter    : Massimo Cetra [off-list ref]
Date        : 2009-10-08 14:51 (4 days old)
References    : http://marc.info/?l=linux-kernel&m=125501488220358&w=4
    

I don't know what to do about this one.

The user indicates that they have the vserver patches applied,
so maybe there is some interaction with that stuff.
  

Actually i found another oops which is very similar to the previous one.
Here, vserver is not involved, and the problem starts at drbd which
lives in kernel space (the other oops started at ocfs2).

Both ocfs2 and drbd make heavy use of network I/O so i guess the problem
is something in the network layer.

Anything i can do to help to debugging and solving this issue ?

Thanks
Max

Problem is kfree_skb() is called from irq context, wich is not allowed.

static void skb_release_head_state(struct sk_buff *skb)
{
...
if (skb->destructor) {
	WARN_ON(in_irq());
	skb->destructor();
}
...
}

virtio_net start_xmit() function calls free_old_xmit_skbs() and 
free_old_xmit_skbs() ultimately calls kfree_skb()

Quick fix would be to use dev_kfree_skb_any() instead,
because netpoll can definitly calls start_xmit()
with irq disabled.

Could you please following patch ?

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 8d00976..54bf091 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c

@@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi)
 		vi->dev->stats.tx_bytes += skb->len;
 		vi->dev->stats.tx_packets++;
 		tot_sgs += skb_vnet_hdr(skb)->num_sg;
-		kfree_skb(skb);
+		dev_kfree_skb_any(skb);
 	}
 	return tot_sgs;
 }

From: Massimo Cetra <redacted>
Date: Tue, 13 Oct 2009 11:11:17 +0200

Here, vserver is not involved, and the problem starts at drbd which
lives in kernel space (the other oops started at ocfs2).

Both ocfs2 and drbd make heavy use of network I/O so i guess the
problem is something in the network layer.

Anything i can do to help to debugging and solving this issue ?

Is all of your traffic going over virtio_net?

From: Eric Dumazet <redacted>
Date: Tue, 13 Oct 2009 12:19:51 +0200

Quick fix would be to use dev_kfree_skb_any() instead,
because netpoll can definitly calls start_xmit()
with irq disabled.

Indeed, because of netpoll(), that is something drivers
must be able to cope with.

Could you please following patch ?

Indeed, let us know how Eric's patch works.

Thanks Eric!

Eric Dumazet ha scritto:

Thank you very much.

Compiling.
It' will be in production in a few minutes.
I'll let you know if the problem arises again.
give me a couple of days because this problem is randomly triggered.
Sometimes it happens multiple times a day, sometimes none.

Max

David Miller ha scritto:

From: Massimo Cetra <redacted>
Date: Tue, 13 Oct 2009 11:11:17 +0200

  

quoted

Here, vserver is not involved, and the problem starts at drbd which
lives in kernel space (the other oops started at ocfs2).

Both ocfs2 and drbd make heavy use of network I/O so i guess the
problem is something in the network layer.

Anything i can do to help to debugging and solving this issue ?
    

Is all of your traffic going over virtio_net?
  

Yes, indeed.
I'm compiling the patch and i'll let you know.

Thanks,
Max

Massimo Cetra ha scritto:

Eric Dumazet ha scritto:

quoted

Could you please following patch ?

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 8d00976..54bf091 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c

@@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct 

virtnet_info *vi)
         vi->dev->stats.tx_bytes += skb->len;
         vi->dev->stats.tx_packets++;
         tot_sgs += skb_vnet_hdr(skb)->num_sg;
-        kfree_skb(skb);
+        dev_kfree_skb_any(skb);
     }
     return tot_sgs;
 }


  

Thank you very much.

Compiling.
It' will be in production in a few minutes.
I'll let you know if the problem arises again.
give me a couple of days because this problem is randomly triggered.
Sometimes it happens multiple times a day, sometimes none.

Eric,
thanks for the patch.
The problem didn't arise again and i haven't seen any warning like that 
on both servers where that problem was happening more frequently.

I would say that it's fixed and if it's not, i'll let you know as soon 
as it happens again.

Thanks again,
Max

Massimo Cetra a écrit :

Eric,
thanks for the patch.
The problem didn't arise again and i haven't seen any warning like that
on both servers where that problem was happening more frequently.

I would say that it's fixed and if it's not, i'll let you know as soon
as it happens again.

Thanks Massimo, I think patch is reasonably safe and should be taken as is :


[PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()

Because netpoll can call netdevice start_xmit() method with
irqs disabled, drivers should not call kfree_skb() from
their start_xmit(), but use dev_kfree_skb_any() instead.

Oct  8 11:16:52 172.30.1.31 [113074.791813] ------------[ cut here ]------------
Oct  8 11:16:52 172.30.1.31 [113074.791813] WARNING: at net/core/skbuff.c:398 \
                skb_release_head_state+0x64/0xc8()
Oct  8 11:16:52 172.30.1.31 [113074.791813] Hardware name: 
Oct  8 11:16:52 172.30.1.31 [113074.791813] Modules linked in: netconsole ocfs2 jbd2 quota_tree \
ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs crc32c drbd cn loop \
serio_raw psmouse snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net pcspkr parport_pc parport \
i2c_piix4 i2c_core button processor evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot \
dm_mod ide_cd_mod cdrom ata_generic ata_piix virtio_blk libata scsi_mod piix ide_pci_generic ide_core \
                virtio_pci virtio_ring virtio floppy thermal fan thermal_sys [last unloaded: netconsole]
Oct  8 11:16:52 172.30.1.31 [113074.791813] Pid: 11132, comm: php5-cgi Tainted: G        W  \
                2.6.31.2-vserver #1
Oct  8 11:16:52 172.30.1.31 [113074.791813] Call Trace:
Oct  8 11:16:52 172.30.1.31 [113074.791813] <IRQ>  [<ffffffff81253cd5>] ? \
                skb_release_head_state+0x64/0xc8
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa01cb139>] ? free_old_xmit_skbs+0x51/0x6e \
                [virtio_net]
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa01cbc85>] ? start_xmit+0x26/0xf2 [virtio_net]
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffffa0429216>] ? write_msg+0x90/0xeb [netconsole]
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8106b142>] ? vx_update_load+0x18/0x13e
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81308309>] ? printk+0x4e/0x5d
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
Oct  8 11:16:52 172.30.1.31 [113074.791813] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
Oct  8 11:16:52 172.30.1.31 [113074.791813] <EOI>  [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31

Reported-and-tested-by: Massimo Cetra <redacted>
Signed-off-by: Eric Dumazet <redacted>
Bug-Entry: http://bugzilla.kernel.org/show_bug.cgi?id=14378
---

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 8d00976..54bf091 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c

@@ -454,7 +454,7 @@ static unsigned int free_old_xmit_skbs(struct virtnet_info *vi)
 		vi->dev->stats.tx_bytes += skb->len;
 		vi->dev->stats.tx_packets++;
 		tot_sgs += skb_vnet_hdr(skb)->num_sg;
-		kfree_skb(skb);
+		dev_kfree_skb_any(skb);
 	}
 	return tot_sgs;
 }

From: Eric Dumazet <redacted>
Date: Thu, 15 Oct 2009 02:36:43 +0200

Massimo Cetra a écrit :

quoted

Eric,
thanks for the patch.
The problem didn't arise again and i haven't seen any warning like that
on both servers where that problem was happening more frequently.

I would say that it's fixed and if it's not, i'll let you know as soon
as it happens again.

Thanks Massimo, I think patch is reasonably safe and should be taken as is :


[PATCH] virtio_net: use dev_kfree_skb_any() in free_old_xmit_skbs()

Applied, thanks Eric.