Re: [PATCH V3 0/4] vDPA/ifcvf: implement shared IRQ feature
From: Zhu, Lingshan <hidden>
Date: 2022-01-27 08:27:58
On 1/27/2022 12:31 PM, Zhu, Lingshan wrote:
On 1/26/2022 10:14 PM, Michael S. Tsirkin wrote:quoted
On Wed, Jan 26, 2022 at 08:49:08PM +0800, Zhu Lingshan wrote:quoted
It has been observed that on some platforms/devices, there may not be enough MSI vectors for virtqueues and the config change. Under such circumstances, the interrupt sources of a device have to share vectors/IRQs. This series implemented a shared IRQ feature for ifcvf. Please help review.Given the history, can you please report which tests were performed with this patchset? Which configs tested? Thanks?Hi Michael, It is ping and netperf tests, and I have set nvectors = 1 and 2 in ifcvf_request_irq(), after ifcvf_alloc_vectors(), to hard coded the number of the allocate vectors. Thanks, Zhu Lingshan
We can verify the tests result by checking the requested IRQs for the two VMs(one vhost-vdpa device per VM). (1)when setting nvectors = 1, only one IRQs requested per VM/device, all vqs and the config interrupt share this IRQ. [lszhu@cra01infra01 ~]$ cat /proc/interrupts | grep ifc 241: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 45 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534528-edge ifcvf[0000:01:00.5]-dev-shared-irq 251: 0 0 0 0 0 0 0 0 0 0 0 41 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 536576-edge ifcvf[0000:01:00.6]-dev-shared-irq (2)when setting nvectors = 2, two IRQs requested for each VM/device, one for all vqs, the other for the config interrupt [lszhu@cra01infra01 ~]$ cat /proc/interrupts | grep ifc 241: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 39 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534528-edge ifcvf[0000:01:00.5]-vqs-shared-irq 242: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534529-edge ifcvf[0000:01:00.5]-config 251: 0 0 0 0 0 0 0 0 0 0 0 39 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 536576-edge ifcvf[0000:01:00.6]-vqs-shared-irq 252: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 536577-edge ifcvf[0000:01:00.6]-config (3)when remove nvectors hardcode, the driver allocates enough vectors for the queues and config interrupt, and we do see better performance because irq_bypass is enabled:(too many lines, cut off) [lszhu@cra01infra01 linux]$ cat /proc/interrupts | grep ifc 241: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534528-edge ifcvf[0000:01:00.5]-0 242: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534529-edge ifcvf[0000:01:00.5]-1 243: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534530-edge ifcvf[0000:01:00.5]-2 244: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534531-edge ifcvf[0000:01:00.5]-3 245: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534532-edge ifcvf[0000:01:00.5]-4 246: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534533-edge ifcvf[0000:01:00.5]-5 247: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534534-edge ifcvf[0000:01:00.5]-6 248: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534535-edge ifcvf[0000:01:00.5]-7 249: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534536-edge ifcvf[0000:01:00.5]-8 250: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534537-edge ifcvf[0000:01:00.5]-config 251: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 536576-edge ifcvf[0000:01:00.6]-0 252: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 536577-edge ifcvf[0000:01:00.6]-1 Thanks, Zhu Lingshan
quoted
quoted
Changes from V2: (1) Fix misuse of nvectors(in ifcvf_alloc_vectors return value)(Michael) (2) Fix misuse of irq = get_vq_irq() in setup irqbypass(Michael) (3) Coding style improvements(Michael) (4) Better naming of device shared irq/shared vq irq Changes from V1: (1) Enable config interrupt when only one vector is allocated(Michael) (2) Clean vectors/IRQs if failed to request config interrupt since config interrupt is a must(Michael) (3) Keep local vdpa_ops, disable irq_bypass by setting IRQ = -EINVAL for shared IRQ case(Michael) (4) Improvements on error messages(Michael) (5) Squash functions implementation patches to the callers(Michael) Zhu Lingshan (4): vDPA/ifcvf: implement IO read/write helpers in the header file vDPA/ifcvf: implement device MSIX vector allocator vhost_vdpa: don't setup irq offloading when irq_num < 0 vDPA/ifcvf: implement shared IRQ feature drivers/vdpa/ifcvf/ifcvf_base.c | 67 +++----- drivers/vdpa/ifcvf/ifcvf_base.h | 60 +++++++- drivers/vdpa/ifcvf/ifcvf_main.c | 260 ++++++++++++++++++++++++++++---- drivers/vhost/vdpa.c | 4 + 4 files changed, 312 insertions(+), 79 deletions(-) -- 2.27.0