RE: Sharing VF device among Xen VMs

-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
On Behalf Of Satish Chowdhury
Sent: Saturday, November 14, 2009 11:50 AM
To: netdev@vger.kernel.org
Subject: Sharing VF device among Xen VMs

Hi,

I am trying to verify a situation where VMs share a VF device of Intel
82576 dual port for data traffic.

Setup:
Case -1: On a Vt-d machine Xen(xen-1) is installed. On dom0 multiple
VFs are created for 82576ET dual port card. One of the VF is
pass-through to a VM.  Now, the VM again has Xen (xen-2) installed.
So, the VMs of Xen-2 have to share the passthrough VF device for data
traffic.

Will I be able to send data from Xen-2 VMs to external world?

I had issues while creating VMs for Xen2. So, could not do the
experiment.

Case-2:  On Xen-1 itself I loaded igbvf driver. Changed xen
configuration to make VF as default interface on dom0. Now VMs of
Xen-1 should share the VF device.

Ping from VF interface on VM  and PF ip address works.
Ping between VMs goes through.
But, ping from domU to another machine on same network on switch doesn't
work.

The arp broadcast request goes out through VF interface. But the arp
reply doesn't reach VF interface, they get routed to PF interface. If
PF interface to the bridge on dom0 then ping from VMs to external
machine work.

In my experiment, the arp reply that reaches the NIC, has mac address
of interface on VM(domU). 82576 performs L2 filtering based on VF MAC
address. So, packet is not queued to VF interface.

Is it possible to add VMs mac address to L2 filtering pool of the NIC?

Regards,
-Satish
--

[Rose, Gregory V] 

No, there is no tool for this right now.  Extensions to the iproute2 package are under review internally and either a set of proposed patches or an RFC should be forthcoming soon.

You are correct that you need to be able to add the MAC addresses of VMs using the bridge interface to the L2 filter table of the NIC before this setup will work correctly.

For now the 82576 in SR-IOV modes of operation only supports communication among the VFs.  You should not try to use the physical function as a bridge interface until the tool I mentioned above becomes available.

- Greg