From: Chris Friesen <hidden> Date: 2008-09-23 15:32:29
We've recently run into an interesting issue, and I was hoping to get
some suggestions.
We've got a boot server configured with two bonded links using the XOR
tx mode. Another board comes up. It's got two links which will
eventually be bonded. It issues a DHCP request to the server over one
of the links, which arrives on one of the bonded slaves. The server
sends a reply, but the XOR results in the packet being sent back via the
other slave. On the booting blade the xid in the packet doesn't match
the xid for the device on which the packet was received, so the packet
is dropped.
What's the proper solution here? Should dhcpd be forcing reply packets
out the slave on which the packet was received?
Chris
From: Jay Vosburgh <hidden> Date: 2008-09-23 16:42:52
Chris Friesen [off-list ref] wrote:
We've got a boot server configured with two bonded links using the XOR tx
mode. Another board comes up. It's got two links which will eventually
be bonded. It issues a DHCP request to the server over one of the links,
which arrives on one of the bonded slaves. The server sends a reply, but
the XOR results in the packet being sent back via the other slave. On the
booting blade the xid in the packet doesn't match the xid for the device
on which the packet was received, so the packet is dropped.
Presumably the blade is matching xid on a per-interface basis.
I'm guessing your problem is really due to the nature of the
intra-chassis network on the blade system. If it's like the ones I'm
familiar with (eth0 of all blades to a single switch, eth1 of all blades
to a different switch, etc), then you can't configure the switches into
an etherchannel group as can be done with the usual configuration
(multiple ports on one switch). If the other blades in the chassis
aren't also configured for etherchannel (balance-xor is an etherchannel
compatible mode), then they'll see the same MAC arriving on multiple
interfaces and not really be prepared to deal with it.
What's the proper solution here? Should dhcpd be forcing reply packets
out the slave on which the packet was received?
Why is the blade issuing DHCP before the bond is configured?
I'm unsure as to whether dhcpd should force replies as
suggested, but recent changes to bonding/net core would probably permit
dhcpd to run directly on the slave (it might need tweaking, I'm not
sure) and accomplish that result. I don't think it would work on older
versions of bonding, since there's no way to tell which slave a packet
arrived on.
In that case, dhcpd would have to run on all slaves, since a
bond that uses DHCP for its address assignment might use any of its
interfaces to issue the request, and it would accept a response on any
interface.
-J
---
-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com
From: Chris Friesen <hidden> Date: 2008-09-23 17:17:43
Jay Vosburgh wrote:
Chris Friesen [off-list ref] wrote:
quoted
On the
booting blade the xid in the packet doesn't match the xid for the device
on which the packet was received, so the packet is dropped.
Presumably the blade is matching xid on a per-interface basis.
Yes, the dhcp code in the kernel does this.
I'm guessing your problem is really due to the nature of the
intra-chassis network on the blade system. If it's like the ones I'm
familiar with (eth0 of all blades to a single switch, eth1 of all blades
to a different switch, etc), then you can't configure the switches into
an etherchannel group as can be done with the usual configuration
(multiple ports on one switch).
This has been handled already. The two switches are a single
etherchannel group and all blades (except the booting one) have
etherchannel enabled.
quoted
What's the proper solution here? Should dhcpd be forcing reply packets
out the slave on which the packet was received?
Why is the blade issuing DHCP before the bond is configured?
The blade is net-booting. The BIOS does DHCP to download the kernel,
and this works. The kernel comes up and does DHCP to obtain an IP
address so it can download the boot image, and this fails as described.
The etherchannel/bond link isn't set up until userspace is loaded and
the bonding driver is insmodded (under multiple aliases with different
sets of slaves).
I'm unsure as to whether dhcpd should force replies as suggested, but
recent changes to bonding/net core would probably permit dhcpd to run
directly on the slave (it might need tweaking, I'm not sure) and
accomplish that result. I don't think it would work on older versions
of bonding, since there's no way to tell which slave a packet arrived
on.
That's not a problem. We've got an older version, but support for
querying which slave a packet arrived on has been added.
Chris
From: Jay Vosburgh <hidden> Date: 2008-09-23 18:18:37
Chris Friesen [off-list ref] wrote:
Jay Vosburgh wrote:
quoted
Chris Friesen [off-list ref] wrote:
[...]
Yes, the dhcp code in the kernel does this.
Ah, I didn't know that.
quoted
I'm guessing your problem is really due to the nature of the
intra-chassis network on the blade system. If it's like the ones I'm
familiar with (eth0 of all blades to a single switch, eth1 of all blades
to a different switch, etc), then you can't configure the switches into
an etherchannel group as can be done with the usual configuration
(multiple ports on one switch).
This has been handled already. The two switches are a single etherchannel
group and all blades (except the booting one) have etherchannel enabled.
Ok, so your problem (if I'm understanding things correctly) is
really a chicken-and-egg type of deal. Your entire network is all
etherchannel, except for this blade, which needs to connect,
temporarily, in a non-etherchannel manner in order to boot up and become
etherchannel-ified (at which point it'll work).
One possible problem with hacking the dhcpd to send back out on
the receiving interface is that (because of the etherchannel) there's no
general guarantee that the switch paths are symmetrical, particularly if
a port is down somewhere.
That's not a problem. We've got an older version, but support for
querying which slave a packet arrived on has been added.
If you're already rolling your own kernels, would it be easier
to remove the xid interface check from the kernel's dhcp client (i.e.,
accept the DHCP reply as if it had arrived on the proper interface for
its xid)? You could make it into some kind of "netboot_on_etherchannel"
option.
-J
---
-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com
From: Chris Friesen <hidden> Date: 2008-09-24 00:16:12
Jay Vosburgh wrote:
Chris Friesen [off-list ref] wrote:
Ok, so your problem (if I'm understanding things correctly) is
really a chicken-and-egg type of deal. Your entire network is all
etherchannel, except for this blade, which needs to connect,
temporarily, in a non-etherchannel manner in order to boot up and become
etherchannel-ified (at which point it'll work).
Basically, yes.
One possible problem with hacking the dhcpd to send back out on
the receiving interface is that (because of the etherchannel) there's no
general guarantee that the switch paths are symmetrical, particularly if
a port is down somewhere.
True.
If you're already rolling your own kernels, would it be easier
to remove the xid interface check from the kernel's dhcp client (i.e.,
accept the DHCP reply as if it had arrived on the proper interface for
its xid)? You could make it into some kind of "netboot_on_etherchannel"
option.
Yes, this is being considered as well. Initially I was against it due
to the possibility of introducing unexpected behaviour, but making it a
boot argument makes a lot of sense, especially since it would allow for
more robust behaviour on certain fault scenarios.
Thanks,
Chris