From: Ben Hutchings <hidden> Date: 2011-05-10 02:38:52
On Fri, 2011-05-06 at 13:12 -0700, Noah Meyerhans wrote:
Package: linux-2.6
Version: 2.6.38-3
Severity: normal
Hi. I've got a system that hosts several kvm virtual hosts. The VMs
access the network via tap devices bridged with a physical interface.
After upgrading to linux-image-2.6.38-2-amd64_2.6.38-4, I noticed that
the virtualhosts were not autoconfiguring their IPv6 interfaces.
Debugging revealed that no multicast was passing over the bridge.
The bridge configuration is:
bridge name bridge id STP enabled interfaces
br0 8000.0002e3080eb5 no eth1
tap0
tap1
tap2
If I attach tcpdump to br0, I can see multicast (e.g. IPv6 Neighbor
Solicitation) packets. However, if I attach tcpdump to eth1, I do not
see multicast packets sourced from one of the VMs.
Downgrading to 2.6.38-3 solves the problem.
This is pretty weird. Debian version 2.6.38-3 has a few bridging
changes from stable 2.6.38.3 and 2.6.38.4, but they don't look like they
would cause this.
Ben.
--
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
From: Stephen Hemminger <hidden> Date: 2011-05-10 03:15:30
On Tue, 10 May 2011 03:38:44 +0100
Ben Hutchings [off-list ref] wrote:
On Fri, 2011-05-06 at 13:12 -0700, Noah Meyerhans wrote:
quoted
Package: linux-2.6
Version: 2.6.38-3
Severity: normal
Hi. I've got a system that hosts several kvm virtual hosts. The VMs
access the network via tap devices bridged with a physical interface.
After upgrading to linux-image-2.6.38-2-amd64_2.6.38-4, I noticed that
the virtualhosts were not autoconfiguring their IPv6 interfaces.
Debugging revealed that no multicast was passing over the bridge.
The bridge configuration is:
bridge name bridge id STP enabled interfaces
br0 8000.0002e3080eb5 no eth1
tap0
tap1
tap2
If I attach tcpdump to br0, I can see multicast (e.g. IPv6 Neighbor
Solicitation) packets. However, if I attach tcpdump to eth1, I do not
see multicast packets sourced from one of the VMs.
Downgrading to 2.6.38-3 solves the problem.
This is pretty weird. Debian version 2.6.38-3 has a few bridging
changes from stable 2.6.38.3 and 2.6.38.4, but they don't look like they
would cause this.
Ben.
There are two possible explainations:
1. In 2.6.37 and kernels the bridge uses IGMP snooping, there were several
fixes to that in the stable kernel; especially related to IPv6.
2. There was also a recent change to block link local multicast
address. But that should impact what you are doing.
On Tue, May 10, 2011 at 03:38:44AM +0100, Ben Hutchings wrote:
This is pretty weird. Debian version 2.6.38-3 has a few bridging
changes from stable 2.6.38.3 and 2.6.38.4, but they don't look like they
would cause this.
I have apparently filed the bug against the wrong version of Debian's
kernel. 2.6.38-3 is not affected, and works as expected. The change
was introduced in -4. That may have been clear from the report itself,
but the report was filed against -3. I've fixed that in the BTS.
I've also confirmed that -5 is affected, to no great surprise.
I'll investigate further.
noah
From: Ben Hutchings <hidden> Date: 2011-05-10 12:43:00
On Mon, 2011-05-09 at 21:38 -0700, Noah Meyerhans wrote:
On Tue, May 10, 2011 at 03:38:44AM +0100, Ben Hutchings wrote:
quoted
This is pretty weird. Debian version 2.6.38-3 has a few bridging
changes from stable 2.6.38.3 and 2.6.38.4, but they don't look like they
would cause this.
I have apparently filed the bug against the wrong version of Debian's
kernel. 2.6.38-3 is not affected, and works as expected. The change
was introduced in -4. That may have been clear from the report itself,
but the report was filed against -3. I've fixed that in the BTS.
I gathered that, and then made the same mistake in writing the above!
The version with the regression, 2.6.38-4, includes the changes from
stable 2.6.38.3 and 2.6.38.4
Ben.
I've also confirmed that -5 is affected, to no great surprise.
I'll investigate further.
noah
--
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
On Mon, 2011-05-09 at 21:38 -0700, Noah Meyerhans wrote:
quoted
On Tue, May 10, 2011 at 03:38:44AM +0100, Ben Hutchings wrote:
quoted
This is pretty weird. Debian version 2.6.38-3 has a few bridging
changes from stable 2.6.38.3 and 2.6.38.4, but they don't look like they
would cause this.
I have apparently filed the bug against the wrong version of Debian's
kernel. 2.6.38-3 is not affected, and works as expected. The change
was introduced in -4. That may have been clear from the report itself,
but the report was filed against -3. I've fixed that in the BTS.
I gathered that, and then made the same mistake in writing the above!
The version with the regression, 2.6.38-4, includes the changes from
stable 2.6.38.3 and 2.6.38.4
Ben.
We just hitted the very same problem yesterday, with debian too , BUT
with vanilla kernels (hand compiled from GIT, with a specific config).
I can confirm that 2.6.38.5 & 2.6.38.6 have this problem, but 2.6.38.3
works fine.
So this is not a debian specific bug as far as I can see.
Cheers,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr
On Tue, May 10, 2011 at 01:42:49PM +0100, Ben Hutchings wrote:
quoted
quoted
This is pretty weird. Debian version 2.6.38-3 has a few bridging
changes from stable 2.6.38.3 and 2.6.38.4, but they don't look like they
would cause this.
I have apparently filed the bug against the wrong version of Debian's
kernel. 2.6.38-3 is not affected, and works as expected. The change
was introduced in -4. That may have been clear from the report itself,
but the report was filed against -3. I've fixed that in the BTS.
I gathered that, and then made the same mistake in writing the above!
The version with the regression, 2.6.38-4, includes the changes from
stable 2.6.38.3 and 2.6.38.4
With a little help from git bisect, I've tracked this regression down to
the following commit to the stable-2.6.38.y tree:
commit 5f1c356a3fadc0c19922d660da723b79bcc9aad7
Author: Herbert Xu [off-list ref]
Date: Fri Mar 18 05:27:28 2011 +0000
bridge: Reset IPCB when entering IP stack on NF_FORWARD
[ Upstream commit 6b1e960fdbd75dcd9bcc3ba5ff8898ff1ad30b6e ]
Whenever we enter the IP stack proper from bridge netfilter we
need to ensure that the skb is in a form the IP stack expects
it to be in.
The entry point on NF_FORWARD did not meet the requirements of
the IP stack, therefore leading to potential crashes/panics.
This patch fixes the problem.
Signed-off-by: Herbert Xu [off-list ref]
Acked-by: Stephen Hemminger [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Signed-off-by: Greg Kroah-Hartman [off-list ref]
The diff is
@@ -741,6 +741,9 @@ static unsigned int br_nf_forward_ip(unsigned int
hook, struct sk_buff *skb,
nf_bridge->mask |= BRNF_PKT_TYPE;
}
+ if (br_parse_ip_options(skb))
+ return NF_DROP;
+
/* The physdev module checks on this */
nf_bridge->mask |= BRNF_BRIDGED;
nf_bridge->physoutdev = skb->dev;
If I revert this change, network connectivity functions as expected for
the VMs on this host.
I don't know enough about this change or the problem it was supposed to
solve to be able to guess about what's going wrong.
noah
From: Stephen Hemminger <hidden> Date: 2011-05-10 22:12:07
On Tue, 10 May 2011 11:05:40 -0700
Noah Meyerhans [off-list ref] wrote:
quoted hunk
On Tue, May 10, 2011 at 01:42:49PM +0100, Ben Hutchings wrote:
quoted
quoted
quoted
This is pretty weird. Debian version 2.6.38-3 has a few bridging
changes from stable 2.6.38.3 and 2.6.38.4, but they don't look like they
would cause this.
I have apparently filed the bug against the wrong version of Debian's
kernel. 2.6.38-3 is not affected, and works as expected. The change
was introduced in -4. That may have been clear from the report itself,
but the report was filed against -3. I've fixed that in the BTS.
I gathered that, and then made the same mistake in writing the above!
The version with the regression, 2.6.38-4, includes the changes from
stable 2.6.38.3 and 2.6.38.4
With a little help from git bisect, I've tracked this regression down to
the following commit to the stable-2.6.38.y tree:
commit 5f1c356a3fadc0c19922d660da723b79bcc9aad7
Author: Herbert Xu [off-list ref]
Date: Fri Mar 18 05:27:28 2011 +0000
bridge: Reset IPCB when entering IP stack on NF_FORWARD
[ Upstream commit 6b1e960fdbd75dcd9bcc3ba5ff8898ff1ad30b6e ]
Whenever we enter the IP stack proper from bridge netfilter we
need to ensure that the skb is in a form the IP stack expects
it to be in.
The entry point on NF_FORWARD did not meet the requirements of
the IP stack, therefore leading to potential crashes/panics.
This patch fixes the problem.
Signed-off-by: Herbert Xu [off-list ref]
Acked-by: Stephen Hemminger [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Signed-off-by: Greg Kroah-Hartman [off-list ref]
The diff is
@@ -741,6 +741,9 @@ static unsigned int br_nf_forward_ip(unsigned int
hook, struct sk_buff *skb,
nf_bridge->mask |= BRNF_PKT_TYPE;
}
+ if (br_parse_ip_options(skb))
+ return NF_DROP;
+
/* The physdev module checks on this */
nf_bridge->mask |= BRNF_BRIDGED;
nf_bridge->physoutdev = skb->dev;
If I revert this change, network connectivity functions as expected for
the VMs on this host.
I don't know enough about this change or the problem it was supposed to
solve to be able to guess about what's going wrong.
noah
There were two more follow on commits in stable related to this.
I recommend merging 2.6.38.6 which includes these.
--
On Tue, May 10, 2011 at 03:11:00PM -0700, Stephen Hemminger wrote:
There were two more follow on commits in stable related to this.
I recommend merging 2.6.38.6 which includes these.
The problem still exists in the current 2.6.38.6. Backing out 5f1c356a
still solves the problem there.
I have not yet tried anything outside the stable-2.6.38.y tree, but it
seems like these same changes are present there, and it's unlikely that
other releases will work any better.
noah
On Tue, May 10, 2011 at 03:11:00PM -0700, Stephen Hemminger wrote:
quoted
There were two more follow on commits in stable related to this.
I recommend merging 2.6.38.6 which includes these.
The problem still exists in the current 2.6.38.6. Backing out 5f1c356a
still solves the problem there.
I have not yet tried anything outside the stable-2.6.38.y tree, but it
seems like these same changes are present there, and it's unlikely that
other releases will work any better.
So the issue is that if we back out that change, we get crashes.
Aparently there is a code path where whatever is existing in the
SKB ip options block matters, and needs to be maintained.
Someone needs to audit all of this and figure out how to fix the
problem properly.
On Tue, May 10, 2011 at 03:11:00PM -0700, Stephen Hemminger wrote:
quoted
There were two more follow on commits in stable related to this.
I recommend merging 2.6.38.6 which includes these.
The problem still exists in the current 2.6.38.6. Backing out 5f1c356a
still solves the problem there.
I have not yet tried anything outside the stable-2.6.38.y tree, but it
seems like these same changes are present there, and it's unlikely that
other releases will work any better.
So the issue is that if we back out that change, we get crashes.
Aparently there is a code path where whatever is existing in the
SKB ip options block matters, and needs to be maintained.
Someone needs to audit all of this and figure out how to fix the
problem properly.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
I suspect tuntap is part of the problem. The skb may not be
allocated with enough padding or something like that. No guarantees
but will do some investigation.
--
From: Stephen Hemminger <hidden> Date: 2011-05-12 23:43:25
On Tue, 10 May 2011 16:35:40 -0700
Noah Meyerhans [off-list ref] wrote:
On Tue, May 10, 2011 at 03:11:00PM -0700, Stephen Hemminger wrote:
quoted
There were two more follow on commits in stable related to this.
I recommend merging 2.6.38.6 which includes these.
The problem still exists in the current 2.6.38.6. Backing out 5f1c356a
still solves the problem there.
I have not yet tried anything outside the stable-2.6.38.y tree, but it
seems like these same changes are present there, and it's unlikely that
other releases will work any better.
Does this fix the problem? The tap driver allocates an skb and throws
it into the receive path, but the skb does not have the same padding
as normal skb's received.
On Thu, May 12, 2011 at 04:43:22PM -0700, Stephen Hemminger wrote:
quoted hunk
quoted
quoted
There were two more follow on commits in stable related to this.
I recommend merging 2.6.38.6 which includes these.
The problem still exists in the current 2.6.38.6. Backing out 5f1c356a
still solves the problem there.
I have not yet tried anything outside the stable-2.6.38.y tree, but it
seems like these same changes are present there, and it's unlikely that
other releases will work any better.
Does this fix the problem? The tap driver allocates an skb and throws
it into the receive path, but the skb does not have the same padding
as normal skb's received.
From: Stephen Hemminger <hidden> Date: 2011-05-13 18:04:26
On Tue, 10 May 2011 16:35:40 -0700
Noah Meyerhans [off-list ref] wrote:
On Tue, May 10, 2011 at 03:11:00PM -0700, Stephen Hemminger wrote:
quoted
There were two more follow on commits in stable related to this.
I recommend merging 2.6.38.6 which includes these.
The problem still exists in the current 2.6.38.6. Backing out 5f1c356a
still solves the problem there.
I have not yet tried anything outside the stable-2.6.38.y tree, but it
seems like these same changes are present there, and it's unlikely that
other releases will work any better.
noah
Is this unique to the tap interfaces or does bridging multicast
not work for all devices?
--
From: Stephen Hemminger <hidden> Date: 2011-05-13 19:53:17
The commit 6b1e960fdbd75dcd9bcc3ba5ff8898ff1ad30b6e
bridge: Reset IPCB when entering IP stack on NF_FORWARD
broke forwarding of IPV6 packets in bridge because it would
call bp_parse_ip_options with an IPV6 packet.
Reported-by: Noah Meyerhans <redacted>
Signed-off-by: Stephen Hemminger <redacted>
---
Patch against net-next-2.6 but must be applied to net-2.6
and stable as well
@@ -737,7 +737,7 @@ static unsigned int br_nf_forward_ip(unsnf_bridge->mask|=BRNF_PKT_TYPE;}-if(br_parse_ip_options(skb))+if(pf==PF_INET&&br_parse_ip_options(skb))returnNF_DROP;/* The physdev module checks on this */
From: Eric Dumazet <hidden> Date: 2011-05-13 20:00:50
Le vendredi 13 mai 2011 à 12:53 -0700, Stephen Hemminger a écrit :
The commit 6b1e960fdbd75dcd9bcc3ba5ff8898ff1ad30b6e
bridge: Reset IPCB when entering IP stack on NF_FORWARD
broke forwarding of IPV6 packets in bridge because it would
call bp_parse_ip_options with an IPV6 packet.
Reported-by: Noah Meyerhans <redacted>
Signed-off-by: Stephen Hemminger <redacted>
---
Patch against net-next-2.6 but must be applied to net-2.6
and stable as well
Well, stable is not needed, since faulty commit is not in 2.6.38
Reviewed-by: Eric Dumazet <redacted>
From: David Miller <davem@davemloft.net> Date: 2011-05-13 20:03:25
From: Eric Dumazet <redacted>
Date: Fri, 13 May 2011 22:00:44 +0200
Le vendredi 13 mai 2011 à 12:53 -0700, Stephen Hemminger a écrit :
quoted
The commit 6b1e960fdbd75dcd9bcc3ba5ff8898ff1ad30b6e
bridge: Reset IPCB when entering IP stack on NF_FORWARD
broke forwarding of IPV6 packets in bridge because it would
call bp_parse_ip_options with an IPV6 packet.
Reported-by: Noah Meyerhans <redacted>
Signed-off-by: Stephen Hemminger <redacted>
---
Patch against net-next-2.6 but must be applied to net-2.6
and stable as well
Well, stable is not needed, since faulty commit is not in 2.6.38
Reviewed-by: Eric Dumazet <redacted>
I do need to queue it up for -stable because the faulty commit is
also queued up there :-)
From: David Miller <davem@davemloft.net> Date: 2011-05-13 20:04:20
From: Stephen Hemminger <redacted>
Date: Fri, 13 May 2011 12:53:14 -0700
The commit 6b1e960fdbd75dcd9bcc3ba5ff8898ff1ad30b6e
bridge: Reset IPCB when entering IP stack on NF_FORWARD
broke forwarding of IPV6 packets in bridge because it would
call bp_parse_ip_options with an IPV6 packet.
Reported-by: Noah Meyerhans <redacted>
Signed-off-by: Stephen Hemminger <redacted>
---
Patch against net-next-2.6 but must be applied to net-2.6
and stable as well
From: Stephen Hemminger <hidden> Date: 2011-05-13 20:24:32
On Fri, 13 May 2011 16:02:32 -0400 (EDT)
David Miller [off-list ref] wrote:
From: Eric Dumazet <redacted>
Date: Fri, 13 May 2011 22:00:44 +0200
quoted
Le vendredi 13 mai 2011 à 12:53 -0700, Stephen Hemminger a écrit :
quoted
The commit 6b1e960fdbd75dcd9bcc3ba5ff8898ff1ad30b6e
bridge: Reset IPCB when entering IP stack on NF_FORWARD
broke forwarding of IPV6 packets in bridge because it would
call bp_parse_ip_options with an IPV6 packet.
Reported-by: Noah Meyerhans <redacted>
Signed-off-by: Stephen Hemminger <redacted>
---
Patch against net-next-2.6 but must be applied to net-2.6
and stable as well
Well, stable is not needed, since faulty commit is not in 2.6.38
Reviewed-by: Eric Dumazet <redacted>
I do need to queue it up for -stable because the faulty commit is
also queued up there :-)
On Fri, May 13, 2011 at 12:53:14PM -0700, Stephen Hemminger wrote:
quoted hunk
The commit 6b1e960fdbd75dcd9bcc3ba5ff8898ff1ad30b6e
bridge: Reset IPCB when entering IP stack on NF_FORWARD
broke forwarding of IPV6 packets in bridge because it would
call bp_parse_ip_options with an IPV6 packet.
Reported-by: Noah Meyerhans <redacted>
Signed-off-by: Stephen Hemminger <redacted>
---
Patch against net-next-2.6 but must be applied to net-2.6
and stable as well
@@ -737,7 +737,7 @@ static unsigned int br_nf_forward_ip(unsnf_bridge->mask|=BRNF_PKT_TYPE;}-if(br_parse_ip_options(skb))+if(pf==PF_INET&&br_parse_ip_options(skb))returnNF_DROP;/* The physdev module checks on this */
I can confirm that this patch fixes the behavior I've been seeing in the
stable-2.6.38.y tree. Thank you, Stephen!
noah