This time answered with a configuration, that I have tested that works 
on 2.6.26.8. The setup is designed to run under dhcp. (small HPC-cluster).

Jay Vosburgh wrote:

Jesper Krogh [off-list ref] wrote:

quoted

I have something that looks like a regression in bonding between 2.6.26.8
and 2.6.27.6 (I'll try the mid-steps later).

Setup: LACP bond(mode=4,mmimon=100) with 3 NIC's and dhcp on top (static
ip didn't work either).

Problem: The bond doesn't get up after bootup. Subsequence ifdown/ifup
brings it up.

	What exactly does "doesn't get up" mean? 

I cant push any traffic through it.

If you configure with
a static IP, and it doesn't come up, what's in /proc/net/bonding/bond0?

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008) 

 

Bonding Mode: IEEE 802.3ad Dynamic link aggregation 

Transmit Hash Policy: layer2 (0) 

MII Status: up 

MII Polling Interval (ms): 100 

Up Delay (ms): 0 

Down Delay (ms): 0 

 

802.3ad info 

LACP rate: slow 

Active Aggregator Info: 

         Aggregator ID: 1 

         Number of ports: 2 

         Actor Key: 17
         Partner Key: 3008
         Partner Mac Address: 02:04:96:34:88:6a

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1e:68:57:82:b2
Aggregator ID: 1

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1e:68:57:82:b3
Aggregator ID: 1

Slave Interface: eth2
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1e:68:57:82:b0
Aggregator ID: 2

# ifconfig bond0
bond0     Link encap:Ethernet  HWaddr 00:1e:68:57:82:b2
           inet addr:10.194.132.90  Bcast:10.194.133.255  Mask:255.255.254.0
           inet6 addr: fe80::21e:68ff:fe57:82b2/64 Scope:Link
           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
           RX packets:5241 errors:0 dropped:0 overruns:0 frame:0
           TX packets:1314 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:382392 (373.4 KB)  TX bytes:126272 (123.3 KB)



doing ifdown bond0 && ifup bond0 brings it correctly up.

root@quad11:~# ping -c 1 -w 5 -W 5 sal
ping: unknown host sal
root@quad11:~# ifdown bond0 && ifup bond0
root@quad11:~# ping -c 1 -w 5 -W 5 sal
PING sal (10.194.133.13) 56(84) bytes of data.
64 bytes from sal (10.194.133.13): icmp_seq=1 ttl=64 time=0.106 ms

--- sal ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.106/0.106/0.106/0.000 ms
root@quad11:~#

 > When it's broken, does it stay broken if you wait a minute or two?

No. It newer comes up.

quoted

I suspect it it timing related. The interface being configured before it's
ready:
root@quad01:~# dmesg | egrep '(dhc|bond)'
[   12.421963] bonding: MII link monitoring set to 100 ms
[   12.483370] bonding: bond0: enslaving eth0 as a backup interface with
an up link.
[   12.523372] bonding: bond0: enslaving eth1 as a backup interface with
an up link.
[   12.611731] bonding: bond0: enslaving eth2 as a backup interface with a
down link.
[   12.780816] warning: `dhclient3' uses 32-bit capabilities (legacy
support in use)
[   15.720491] bonding: bond0: link status definitely up for interface eth2.
[   87.800324] bond0: no IPv6 routers present

	This looks like one of the slaves (eth2) took longer to assert
carrier up (slower autoneg, perhaps) than the other two (eth0 and eth1).
That wouldn't necessarily cause DHCP to fail; 802.3ad is allowed to
aggregate eth0 and eth1 and use them independently of eth2.

 >

	However, if eth0 and eth1 are incorrectly asserting carrier up
(before autoneg is complete), then that could cause problems.  If that's
the case, then checking /proc/net/bonding/bond0 should show the actual
aggregation status.  If lacp is set to slow (the default), then it
should try to reaggregate 30 seconds later, and that would clear up the
aggregation.  DHCP would still need to restart, though.
	What distro are you using?  I just tried the bonding driver from
the current net-next-2.6 mainline on recent SuSE and 802.3ad + DHCP
works fine for me.  I'm using BCM 5704s (tg3).

quoted

The setup is a 3 NIC bond on a Sun X2200 dual-cpu Quad-core server.
I have similar bond on a X4600 where they works with 2.6.27.6 so I suspect
that the difference is that the X4600 has all NIC's from the
same vendor where as the X2200 has 2 Broadcom NIC's and 2 NVidia nics.

	Which flavor (Broadcom or Nvidia) are the 3 devices that are the
same?

The three NICS are mixed. 2 forcedeth Nvidia(eth0,eth1) and one Tigon3 
(Broadcom) (eth2).

-- 
Jesper