Re: If 1 KVM Guest loads the virtio-pci, on top of dpdkvhostuser OVS socket interface, it slows down everything!

I could reproduce the issue and this can be fixed as below

Firstly, the throughput issues observed with other VMs when a new VM is started can be fixed using the patch in the thread http://openvswitch.org/pipermail/dev/2016-May/071615.html.  I have put up an explanation in this thread for the cause of issue especially with multi VM setup on OVS DPDK. 

On a Multi VM setup even with the above patch applied, one might see aggregate throughput difference when vNIC is bind to igb_uio vs virtio-pci, this is for the fact that the interrupt overhead is significantly higher when virtio-pci is in use. 

More importantly if you have setup explicit flows matching VM's MAC/IP, disabling the flows to the VM that are idle would improve the aggregate throughput and lessen the burden on the pmd thread.   'watch -d ./utilities/ovs-appctl dpctl/show -s' will show no. of packet stats.

Regards,
Bhanu Prakash.

-----Original Message-----
From: dev [mailto:dev-bounces@openvswitch.org] On Behalf Of Christian
Ehrhardt
Sent: Wednesday, May 25, 2016 7:08 AM
To: Martinx - ジェームズ <redacted>
Cc: <redacted> <redacted>; dev <redacted>;
qemu-stable@nongnu.org
Subject: Re: [ovs-dev] If 1 KVM Guest loads the virtio-pci, on top of
dpdkvhostuser OVS socket interface, it slows down everything!

Hi again,
another forgotten case.

I currently I lack the HW to fully reproduce this, but the video summary is
pretty good and shows the issue in an impressive way.

Also the description is good and here as well I wonder if anybody else could
reproduce this.
Any hints / insights are welcome.

P.S. and also again - two list cross posting, but here as well it is yet unclear
which it belongs to so I'll keep it as well

Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

On Sun, May 22, 2016 at 6:35 PM, Martinx - ジェームズ
[off-list ref]
wrote:

quoted

Guys,

 I'm seeing a strange problem here, in my OVS+DPDK deployment, on top
of Ubuntu 16.04 (DPDK 2.2 and OVS 2.5).

 Here is what I'm trying to do: run OVS with DPDK at the host, for KVM
Guests that also, will be running more DPDK Apps.

 The host have 2 x 10G NICs, for OVS+DPDK and each KVM Guest receives
its own VLAN tagged traffic (or all tags).

 There is an IXIA Traffic Generator sending 10G of traffic on both
directions (20G total).

 Exemplifying, the problem is, lets say that I already have 2 VMs (or
10) running DPDK Apps (on top of dpdkvhostuser), everything is working
as expected, then, if I boot the 3rd (or 11) KVM Guest, the OVS+DPDK
bridge at the host, slows down, a lot! The 3rd (or 11) VM affects not
only the host, but also, all the other neighbors VMs!!!

 NOTE: This problem appear since the boot of VM 1.

 Soon as you, inside of the 3rd VM, bind the VirtIO NIC to the
DPDK-Compative Drivers, the speed comes back to normal. If you bind it
back to "virtio-pci", boom! The OVS+DPDK at the host and all VMs loses
too much speed.

 This problem is detailed at the following bug report:

--
The OVS+DPDK dpdkvhostuser socket bridge, only works as expected, if
the KVM Guest also have DPDK drivers loaded:

https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1577256
--

 Also, I've recorded a ~15 min screen cast video about this problem,
so, you guys can see exactly what is happening here.

https://www.youtube.com/v/yHnaSikd9XY?version=3&vq=hd720&autoplay=
1

quoted

 * At 5:25, I'm starting a VM that will boot up and load a DPDK App;

 * At 5:33, OVS+DPDK is messed up, it loses speed;
   The KVM running with virtio-pci drivers breaks OVS+DPDK at the
host;

 * At 6:50, DPDK inside of the KVM guest loads up its drivers, kicking
"virtio-pci", speed back to normal at the host;

 * At 7:43, started another KVM Guest, now, while virtio-pci driver is
running, the OVS+DPDK at the host and the other VM, are very, very
slow;

 * At 8:52, the second VM loads up DPDK Drivers, kicking virtio-pci,
the speed is back to normal at the host, and on the other VM too;

 * At 10:00, the Ubuntu VM loads up virtio-pci drivers on its boot,
the speed dropped at the hosts and on the other VMs;

 * 11:57, I'm starting "service dpdk start" inside of the Ubuntu
guest, to kick up virtio-pci, and bang! Speed is back to normal
everywhere;

 * 12:51, I'm trying to unbind the DPDK Drivers and return the
virtio-pci, I forgot the syntax while recording the video, which is:
"dpdk_nic_bind -b <ID> virtio-pci", so, I just rebooted it. But both
"reboot" or "rebind to virtio-pci" triggers the bug.


NOTE: I tried to subscriber to qemu-devel but, it is not working, I'm
not receiving the confirmation e-mail, while qemu-stable worked. I
don't know if it worth sending it to Linux Kernel too...


Regards,
Thiago

_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev