From: Eric Dumazet <hidden> Date: 2011-02-02 11:25:17
Le mercredi 02 février 2011 à 11:52 +0100, Eric Dumazet a écrit :
Le mercredi 02 février 2011 à 09:53 +0100, Yann Dupont a écrit :
quoted
Hello.
We recently upgraded one machine with vanilla 2.6.37, and experienced 2
kernel oops since. Each oops is after ~1 week of uptime.
The last oops was last night but we didn't had any trace.
Please ignore, this patch was for linux-2.6 tree, 2.6.37 was not
affected by the problem.
So its another problem... Is there anything particular you do on this
machine ?
Le mercredi 02 février 2011 à 11:52 +0100, Eric Dumazet a écrit :
quoted
Le mercredi 02 février 2011 à 09:53 +0100, Yann Dupont a écrit :
quoted
Hello.
We recently upgraded one machine with vanilla 2.6.37, and experienced 2
kernel oops since. Each oops is after ~1 week of uptime.
The last oops was last night but we didn't had any trace.
Please ignore, this patch was for linux-2.6 tree, 2.6.37 was not
affected by the problem.
So its another problem... Is there anything particular you do on this
machine ?
Nothing really special there, we run a lot (20) of KVM guest (mainly
linux firewalls for lots of differents vlan), so we have a lot of
bridges vlan & tun/tap.
Oh, and CONFIG_BRIDGE_IGMP_SNOOPING is set to n (because of the other
bug already sent to netdev - more to come on next mail)
Hard to say if this BUG is new in 2.6.37. This host was running fine
with 2.6.34.2 since August 2010.
Bisecting will be hard due to the time to trigger the bug (and the fact
that this machine is a production machine)
Anyway, I can test with a specific kernel version if you suspect something.
Regards,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr
From: Eric Dumazet <hidden> Date: 2011-02-02 14:53:40
Le mercredi 02 février 2011 à 14:08 +0100, Yann Dupont a écrit :
Le 02/02/2011 12:24, Eric Dumazet a écrit :
quoted
Le mercredi 02 février 2011 à 11:52 +0100, Eric Dumazet a écrit :
quoted
Le mercredi 02 février 2011 à 09:53 +0100, Yann Dupont a écrit :
quoted
Hello.
We recently upgraded one machine with vanilla 2.6.37, and experienced 2
kernel oops since. Each oops is after ~1 week of uptime.
The last oops was last night but we didn't had any trace.
Please ignore, this patch was for linux-2.6 tree, 2.6.37 was not
affected by the problem.
So its another problem... Is there anything particular you do on this
machine ?
Nothing really special there, we run a lot (20) of KVM guest (mainly
linux firewalls for lots of differents vlan), so we have a lot of
bridges vlan & tun/tap.
Oh, and CONFIG_BRIDGE_IGMP_SNOOPING is set to n (because of the other
bug already sent to netdev - more to come on next mail)
Hard to say if this BUG is new in 2.6.37. This host was running fine
with 2.6.34.2 since August 2010.
Bisecting will be hard due to the time to trigger the bug (and the fact
that this machine is a production machine)
Anyway, I can test with a specific kernel version if you suspect something.
I suspect a mem corruption from another layer (not inetpeer)
Unfortunately many kmem caches share the "64 bytes" cache.
Could you please add "slub_nomerge" on your boot command ?
This way, we can separate corruptions on each cache.
On your crash, one inetpeer contain garbage on unused_lists next/prev
pointers :
RCX: 0000000000000005
RDX: 0b000209f1beadde
Definitly something overwrote these values with non pointers values.
Le mercredi 02 février 2011 à 14:08 +0100, Yann Dupont a écrit :
quoted
Le 02/02/2011 12:24, Eric Dumazet a écrit :
quoted
Le mercredi 02 février 2011 à 11:52 +0100, Eric Dumazet a écrit :
quoted
Le mercredi 02 février 2011 à 09:53 +0100, Yann Dupont a écrit :
quoted
Hello.
We recently upgraded one machine with vanilla 2.6.37, and experienced 2
kernel oops since. Each oops is after ~1 week of uptime.
The last oops was last night but we didn't had any trace.
Please ignore, this patch was for linux-2.6 tree, 2.6.37 was not
affected by the problem.
So its another problem... Is there anything particular you do on this
machine ?
Nothing really special there, we run a lot (20) of KVM guest (mainly
linux firewalls for lots of differents vlan), so we have a lot of
bridges vlan& tun/tap.
Oh, and CONFIG_BRIDGE_IGMP_SNOOPING is set to n (because of the other
bug already sent to netdev - more to come on next mail)
Hard to say if this BUG is new in 2.6.37. This host was running fine
with 2.6.34.2 since August 2010.
Bisecting will be hard due to the time to trigger the bug (and the fact
that this machine is a production machine)
Anyway, I can test with a specific kernel version if you suspect something.
I suspect a mem corruption from another layer (not inetpeer)
Unfortunately many kmem caches share the "64 bytes" cache.
Could you please add "slub_nomerge" on your boot command ?
Ok, will do it at 18:30 CET (to minimize impact)
It the suspected bug SLUB related ?
The 2.6.34.2 kernel previously used on that server used SLAB.
2 questions :
-How can I be sure slub_nomerge is active ? Boot message ?
-Is there a very severe impact on performance ?
Regards,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr
I suspect a mem corruption from another layer (not inetpeer)
Unfortunately many kmem caches share the "64 bytes" cache.
Could you please add "slub_nomerge" on your boot command ?
...
quoted
-Is there a very severe impact on performance ?
not at all
Maybe there is an impact after all : since then, we don't have problems
anymore !
linkwood:~# uptime
11:42:03 up 39 days, 17:08, 3 users, load average: 0.01, 0.03, 0.05
So... could slub_nomerge hide or simply avoid the problem ?
Or are we just lucky this time ?
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr
From: Eric Dumazet <hidden> Date: 2011-03-14 13:14:30
Le lundi 14 mars 2011 à 11:44 +0100, Yann Dupont a écrit :
Le 02/02/2011 16:08, Eric Dumazet a écrit :
quoted
I suspect a mem corruption from another layer (not inetpeer)
Unfortunately many kmem caches share the "64 bytes" cache.
Could you please add "slub_nomerge" on your boot command ?
...
quoted
quoted
-Is there a very severe impact on performance ?
not at all
Maybe there is an impact after all : since then, we don't have problems
anymore !
linkwood:~# uptime
11:42:03 up 39 days, 17:08, 3 users, load average: 0.01, 0.03, 0.05
So... could slub_nomerge hide or simply avoid the problem ?
Or are we just lucky this time ?
I would say you are lucky ;)
Not all memory corruptions are noticed. Sometimes it touch unused parts
of memory, or some parts with no critical content.