From: Hannes Frederic Sowa <hidden> Date: 2014-02-03 22:03:24
Hi Ben,
On Mon, Feb 03, 2014 at 12:37:52PM -0800, Ben Greear wrote:
The kernel has some additional patches, but not much to IPv6.
The bug is that when we have lots of mac-vlans on some ixgbe ports
(500 per interface in this case), and boot up the system with the ports unplugged,
we get this crash almost every time. Boot-up is going to do normal bootup
stuff plus create and configure the 1000 mac-vlans, dump their routing
tables, etc.
We are using one routing table per network device, and some
ip rules.
If we plug in the ixgbe ports, we do not ever see a crash.
We have not yet tried reproducing it on other drivers, but I suspect
the issue is not related to ixgbe.
Any ideas on this one?
Could you bring the machine to a panic again with enabling RT6_DEBUG at the
top of ip6_fib.c and send a dump of the trace?
Thanks,
Hannes
From: Ben Greear <hidden> Date: 2014-02-03 22:06:55
On 02/03/2014 02:03 PM, Hannes Frederic Sowa wrote:
Hi Ben,
On Mon, Feb 03, 2014 at 12:37:52PM -0800, Ben Greear wrote:
quoted
The kernel has some additional patches, but not much to IPv6.
The bug is that when we have lots of mac-vlans on some ixgbe ports
(500 per interface in this case), and boot up the system with the ports unplugged,
we get this crash almost every time. Boot-up is going to do normal bootup
stuff plus create and configure the 1000 mac-vlans, dump their routing
tables, etc.
We are using one routing table per network device, and some
ip rules.
If we plug in the ixgbe ports, we do not ever see a crash.
We have not yet tried reproducing it on other drivers, but I suspect
the issue is not related to ixgbe.
Any ideas on this one?
Could you bring the machine to a panic again with enabling RT6_DEBUG at the
top of ip6_fib.c and send a dump of the trace?
Yes, but it will be a bit until we can create a duplicate machine.
We ended up delivering the machine with a note to make sure the
interfaces were plugged in (we found the bug hours before shipping
the system, of course).
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
From: Ben Greear <hidden> Date: 2014-02-08 16:43:33
On 02/03/2014 02:06 PM, Ben Greear wrote:
On 02/03/2014 02:03 PM, Hannes Frederic Sowa wrote:
quoted
Hi Ben,
On Mon, Feb 03, 2014 at 12:37:52PM -0800, Ben Greear wrote:
quoted
The kernel has some additional patches, but not much to IPv6.
The bug is that when we have lots of mac-vlans on some ixgbe ports
(500 per interface in this case), and boot up the system with the ports unplugged,
we get this crash almost every time. Boot-up is going to do normal bootup
stuff plus create and configure the 1000 mac-vlans, dump their routing
tables, etc.
We are using one routing table per network device, and some
ip rules.
If we plug in the ixgbe ports, we do not ever see a crash.
We have not yet tried reproducing it on other drivers, but I suspect
the issue is not related to ixgbe.
Any ideas on this one?
Could you bring the machine to a panic again with enabling RT6_DEBUG at the
top of ip6_fib.c and send a dump of the trace?
Yes, but it will be a bit until we can create a duplicate machine.
We ended up delivering the machine with a note to make sure the
interfaces were plugged in (we found the bug hours before shipping
the system, of course).
According to my system test guy, it took a lot longer to reproduce
the problem with the debug enabled kernel, but I do not see any extra
debug messages on the serial console logging or in /var/log/messages
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
From: Hannes Frederic Sowa <hidden> Date: 2014-02-08 17:23:11
On Sat, Feb 08, 2014 at 08:43:32AM -0800, Ben Greear wrote:
On 02/03/2014 02:06 PM, Ben Greear wrote:
quoted
On 02/03/2014 02:03 PM, Hannes Frederic Sowa wrote:
quoted
Hi Ben,
On Mon, Feb 03, 2014 at 12:37:52PM -0800, Ben Greear wrote:
quoted
The kernel has some additional patches, but not much to IPv6.
The bug is that when we have lots of mac-vlans on some ixgbe ports
(500 per interface in this case), and boot up the system with the ports unplugged,
we get this crash almost every time. Boot-up is going to do normal bootup
stuff plus create and configure the 1000 mac-vlans, dump their routing
tables, etc.
We are using one routing table per network device, and some
ip rules.
If we plug in the ixgbe ports, we do not ever see a crash.
We have not yet tried reproducing it on other drivers, but I suspect
the issue is not related to ixgbe.
Any ideas on this one?
Could you bring the machine to a panic again with enabling RT6_DEBUG at the
top of ip6_fib.c and send a dump of the trace?
Yes, but it will be a bit until we can create a duplicate machine.
We ended up delivering the machine with a note to make sure the
interfaces were plugged in (we found the bug hours before shipping
the system, of course).
According to my system test guy, it took a lot longer to reproduce
the problem with the debug enabled kernel, but I do not see any extra
debug messages on the serial console logging or in /var/log/messages
Sounds like a race, then, like I thought.
I forgot, those are pr_debugs, I usually enable them with
$ echo file net/ipv6/ip6_fib.c +p > /sys/kernel/debug/dynamic_debug/control
RT6_TRACE is pretty noisy so you should see output immediatley if you do ipv6
traffic. Other way is to specify dyndbg="file net/ipv6/ip6_fib.c +p" on the
kernel command line.
Try before doing to play with that until you can confirm the output showes up
on the console.
Thanks again,
Hannes
From: Hannes Frederic Sowa <hidden> Date: 2014-09-28 12:11:38
Hi Hongmei,
On Fri, Sep 26, 2014, at 00:24, Hongmei Li wrote:
Hi Hannes and Ben,
I encounter the same kernel panic.
Do you guys have any progress on this issue?
Unfortunately not, not looked into it for a long time.
Can you send me a fast reproducer script? I have better machines to
reproduce this issue nowadays.
Thanks,
Hannes
From: Hongmei Li <hidden> Date: 2014-09-29 18:15:27
Thanks Hannes for your prompt response!
We just encountered this issue two times in our product stability test so far,
and my panic backtrace is the exactly same with the one reported here.
I don't know how to reproduce the issue till now.
I tried several method, unfortunately, I can not reproduce it by myself. :(
Thanks,
Hongmei
From: Ben Greear <hidden> Date: 2014-09-29 18:44:23
On 09/29/2014 11:15 AM, Hongmei Li wrote:
Thanks Hannes for your prompt response!
We just encountered this issue two times in our product stability test so far,
and my panic backtrace is the exactly same with the one reported here.
I don't know how to reproduce the issue till now.
I tried several method, unfortunately, I can not reproduce it by myself. :(
We could reproduce it easily using our user-space tool that creates 1000
mac-vlans, configures them, dumps routing tables, etc. But, we could
only reproduce it when we had the ixgbe ports unplugged. If they
were properly connected to a switch, we did not see any crashes.
My original email on 2/3/2014 has more details, and the thread that
follows has some info on debugging we did at the time.
We will be happy to test patches...
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
From: Hannes Frederic Sowa <hidden> Date: 2014-09-29 19:12:57
On Mo, 2014-09-29 at 11:44 -0700, Ben Greear wrote:
On 09/29/2014 11:15 AM, Hongmei Li wrote:
quoted
Thanks Hannes for your prompt response!
We just encountered this issue two times in our product stability test so far,
and my panic backtrace is the exactly same with the one reported here.
I don't know how to reproduce the issue till now.
I tried several method, unfortunately, I can not reproduce it by myself. :(
We could reproduce it easily using our user-space tool that creates 1000
mac-vlans, configures them, dumps routing tables, etc. But, we could
only reproduce it when we had the ixgbe ports unplugged. If they
were properly connected to a switch, we did not see any crashes.
My original email on 2/3/2014 has more details, and the thread that
follows has some info on debugging we did at the time.
I just tried to reproduce the problem, disabling the port on a switch
and setting up 1000 macvlans on a ixgbe, enabling ipv6 and dumping
routing tables. Unluckily I had no success. :(
Any more hints?
Bye,
Hannes
From: Ben Greear <hidden> Date: 2014-09-29 19:48:20
On 09/29/2014 12:03 PM, Hannes Frederic Sowa wrote:
On Mo, 2014-09-29 at 11:44 -0700, Ben Greear wrote:
quoted
On 09/29/2014 11:15 AM, Hongmei Li wrote:
quoted
Thanks Hannes for your prompt response!
We just encountered this issue two times in our product stability test so far,
and my panic backtrace is the exactly same with the one reported here.
I don't know how to reproduce the issue till now.
I tried several method, unfortunately, I can not reproduce it by myself. :(
We could reproduce it easily using our user-space tool that creates 1000
mac-vlans, configures them, dumps routing tables, etc. But, we could
only reproduce it when we had the ixgbe ports unplugged. If they
were properly connected to a switch, we did not see any crashes.
My original email on 2/3/2014 has more details, and the thread that
follows has some info on debugging we did at the time.
I just tried to reproduce the problem, disabling the port on a switch
and setting up 1000 macvlans on a ixgbe, enabling ipv6 and dumping
routing tables. Unluckily I had no success. :(
Any more hints?
We are going to be running up to 20 concurrent scripts that
will be dumping routes & ips, and configuring ip addresses
and routes during bringup.
I don't have a simple stand-alone way to reproduce this,
but at least when we reported the problem is was very easy
for us to reproduce with our tool.
Maybe 20 scripts running in parallel that randomly configured and dumped routes
and ip addresses on random interfaces would do the trick?
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
From: Cong Wang <hidden> Date: 2014-09-29 20:39:38
On Mon, Sep 29, 2014 at 12:48 PM, Ben Greear [off-list ref] wrote:
We are going to be running up to 20 concurrent scripts that
will be dumping routes & ips, and configuring ip addresses
and routes during bringup.
I don't have a simple stand-alone way to reproduce this,
but at least when we reported the problem is was very easy
for us to reproduce with our tool.
Maybe 20 scripts running in parallel that randomly configured and dumped routes
and ip addresses on random interfaces would do the trick?
I think the most important question is why is this related with macvlan?
IPv6 is L3 while macvlan pure L2, if this is a IPv6 routing bug, it should
not be limited to macvlan.
What does your IPv6 routing table look like? And how do you configure those
macvlan interfaces?
From: Ben Greear <hidden> Date: 2014-09-29 21:24:55
On 09/29/2014 01:39 PM, Cong Wang wrote:
On Mon, Sep 29, 2014 at 12:48 PM, Ben Greear [off-list ref] wrote:
quoted
We are going to be running up to 20 concurrent scripts that
will be dumping routes & ips, and configuring ip addresses
and routes during bringup.
I don't have a simple stand-alone way to reproduce this,
but at least when we reported the problem is was very easy
for us to reproduce with our tool.
Maybe 20 scripts running in parallel that randomly configured and dumped routes
and ip addresses on random interfaces would do the trick?
I think the most important question is why is this related with macvlan?
IPv6 is L3 while macvlan pure L2, if this is a IPv6 routing bug, it should
not be limited to macvlan.
We saw it using mac-vlans on top of ixgbe. Probably it can be reproduced
elsewhere, but since we had a good test case, we didn't bother trying lots
of other combinations.
Just enabling some debug code caused the problem to be much harder
to reproduce, so it is probably some sort of race. Maybe lots of mac-vlans
make it easier to hit.
What does your IPv6 routing table look like? And how do you configure those
macvlan interfaces?
Each interface would have it's own routing table, with at least as subnet
route. I'm not sure we were adding a default gateway or not.
The main routing table would also have some auto-created routes
I think.
It is configured using 'ip' to add and dump routes.
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
On Mon, Sep 29, 2014 at 5:24 PM, Ben Greear [off-list ref] wrote:
On 09/29/2014 01:39 PM, Cong Wang wrote:
quoted
On Mon, Sep 29, 2014 at 12:48 PM, Ben Greear [off-list ref] wrote:
quoted
We are going to be running up to 20 concurrent scripts that
will be dumping routes & ips, and configuring ip addresses
and routes during bringup.
I don't have a simple stand-alone way to reproduce this,
but at least when we reported the problem is was very easy
for us to reproduce with our tool.
Maybe 20 scripts running in parallel that randomly configured and dumped routes
and ip addresses on random interfaces would do the trick?
I think the most important question is why is this related with macvlan?
IPv6 is L3 while macvlan pure L2, if this is a IPv6 routing bug, it should
not be limited to macvlan.
We saw it using mac-vlans on top of ixgbe. Probably it can be reproduced
elsewhere, but since we had a good test case, we didn't bother trying lots
of other combinations.
Just enabling some debug code caused the problem to be much harder
to reproduce, so it is probably some sort of race. Maybe lots of mac-vlans
make it easier to hit.
quoted
What does your IPv6 routing table look like? And how do you configure those
macvlan interfaces?
Each interface would have it's own routing table, with at least as subnet
route. I'm not sure we were adding a default gateway or not.
The main routing table would also have some auto-created routes
I think.
It is configured using 'ip' to add and dump routes.
Ben
Just saw this thread. Can you check if this commit makes a difference:
commit 40b8fe45d1f094e3babe7b2dc2b71557ab71401d
Author: Vlad Yasevich [off-list ref]
Date: Mon Sep 22 16:34:17 2014 -0400
macvtap: Fix race between device delete and open.
Thanks
-vlad
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Ben Greear <hidden> Date: 2014-10-13 18:06:13
On 10/12/2014 04:42 AM, Vladislav Yasevich wrote:
On Mon, Sep 29, 2014 at 5:24 PM, Ben Greear [off-list ref] wrote:
quoted
On 09/29/2014 01:39 PM, Cong Wang wrote:
quoted
On Mon, Sep 29, 2014 at 12:48 PM, Ben Greear [off-list ref] wrote:
quoted
We are going to be running up to 20 concurrent scripts that
will be dumping routes & ips, and configuring ip addresses
and routes during bringup.
I don't have a simple stand-alone way to reproduce this,
but at least when we reported the problem is was very easy
for us to reproduce with our tool.
Maybe 20 scripts running in parallel that randomly configured and dumped routes
and ip addresses on random interfaces would do the trick?
I think the most important question is why is this related with macvlan?
IPv6 is L3 while macvlan pure L2, if this is a IPv6 routing bug, it should
not be limited to macvlan.
We saw it using mac-vlans on top of ixgbe. Probably it can be reproduced
elsewhere, but since we had a good test case, we didn't bother trying lots
of other combinations.
Just enabling some debug code caused the problem to be much harder
to reproduce, so it is probably some sort of race. Maybe lots of mac-vlans
make it easier to hit.
quoted
What does your IPv6 routing table look like? And how do you configure those
macvlan interfaces?
Each interface would have it's own routing table, with at least as subnet
route. I'm not sure we were adding a default gateway or not.
The main routing table would also have some auto-created routes
I think.
It is configured using 'ip' to add and dump routes.
Ben
Just saw this thread. Can you check if this commit makes a difference:
commit 40b8fe45d1f094e3babe7b2dc2b71557ab71401d
Author: Vlad Yasevich [off-list ref]
Date: Mon Sep 22 16:34:17 2014 -0400
macvtap: Fix race between device delete and open.
We can retest this, but we do not use macvtap, and we have higher-priority
things to test at the moment, so might be a while.
Thanks,
Ben
Thanks
-vlad
quoted
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html