discussion questions: SR-IOV, virtualization, and bonding

Hi all,

I wanted to just highlight some issues that we're seeing and see what 
others are doing in this area.

Our configuration is that we have a host with SR-IOV-capable NICs with 
bonding enabled on the PF.  Depending on the exact system it could be 
active/standby or some form of active/active.

In the guests we generally have several VFs (corresponding to several 
PFs) and we want to bond them for reliability.

We're seeing a number of issues:

1) If the guests use arp monitoring then broadcast arp packets from the 
guests are visible on the other guests and on the host, and can cause 
them to think the link is good even if we aren't receiving arp packets 
from the external network.  (I'm assuming carrier is up.)

2) If both the host and guest use active/backup but pick different 
devices as the active, there is no traffic between host/guest over the 
bond link.  Packets are sent out the active and looped back internally 
to arrive on the inactive, then skb_bond_should_drop() suppresses them.

3) For active/standby the default is to set the standby to the MAC 
address of the bond.  If the host has already set the MAC address (using 
some algorithm to ensure uniqueness within the local network) then the 
guest is not allowed to change it.


So far the solutions to 1 seem to be either using arp validation (which 
currently doesn't exist for loadbalancing modes) or else have the 
underlying ethernet driver distinguish between packets coming from the 
wire vs being looped back internally and have the bonding driver only 
set last_rx for external packets.

For issue 2, it would seem beneficial for the host to be able to ensure 
that the guest uses the same link as the active.  I don't see a tidy 
solution here.  One somewhat messy possibility here is to have bonding 
send a message to the standby PF which then tells all its VFs to fake 
loss of carrier.

For issue 3, the logical solution would seem to be some way of assigning 
a list of "valid" mac addresses to a given VF--like maybe all MAC 
addresses assigned to a VM or something.  Anyone have any bright ideas?


I'm sure we're not the only ones running into this, so what are others 
doing?  Is the only current option to use active/active with miimon?

Chris

-- 

Chris Friesen
Software Designer

3500 Carling Avenue
Ottawa, Ontario K2H 8E9
www.genband.com