On 10/21/21 3:17 PM, zhenwei pi wrote:

On 10/21/21 1:03 PM, Wanpeng Li wrote:

quoted

On Thu, 21 Oct 2021 at 11:05, zhenwei pi [off-list ref] wrote:

quoted

On 10/21/21 4:12 AM, Sean Christopherson wrote:

quoted

On Wed, Oct 20, 2021, Wanpeng Li wrote:

quoted

On Wed, 20 Oct 2021 at 20:08, zhenwei pi [off-list ref] 
wrote:

quoted

Although host side exposes KVM PV SEND IPI feature to guest side,
guest should still have a chance to disable it.

A typicall case of this parameter:
If the host AMD server enables AVIC feature, the flat mode of APIC
get better performance in the guest.

Hmm, I didn't find enough valuable information in your posting. We
observe AMD a lot before.
https://lore.kernel.org/all/CANRm+Cx597FNRUCyVz1D=B6Vs2GX3Sw57X7Muk+yMpi_hb+v1w@mail.gmail.com/T/#u (local) 

I too would like to see numbers.  I suspect the answer is going to 
be that
AVIC performs poorly in CPU overcommit scenarios because of the cost 
of managing
the tables and handling "failed delivery" exits, but that AVIC does 
quite well
when vCPUs are pinned 1:1 and IPIs rarely require an exit to the host.

Test env:
CPU: AMD EPYC 7642 48-Core Processor

Kmod args(enable avic and disable nested):
modprobe kvm-amd nested=0 avic=1 npt=1

QEMU args(disable x2apic):
... -cpu host,x2apic=off ...

Benchmark tool:
https://github.com/bytedance/kvm-utils/tree/master/microbenchmark/apic-ipi 


~# insmod apic_ipi.ko options=5 && dmesg -c

   apic_ipi: 1 NUMA node(s)
   apic_ipi: apic [flat]
   apic_ipi: apic->send_IPI[default_send_IPI_single+0x0/0x40]
   apic_ipi: apic->send_IPI_mask[kvm_send_ipi_mask+0x0/0x10]
   apic_ipi:     IPI[kvm_send_ipi_mask] from CPU[0] to CPU[1]
   apic_ipi:             total cycles 375671259, avg 3756
   apic_ipi:     IPI[flat_send_IPI_mask] from CPU[0] to CPU[1]
   apic_ipi:             total cycles 221961822, avg 2219


apic->send_IPI_mask[kvm_send_ipi_mask+0x0/0x10]
    -> This line show current send_IPI_mask is kvm_send_ipi_mask(because
of PV SEND IPI FEATURE)

apic_ipi:       IPI[kvm_send_ipi_mask] from CPU[0] to CPU[1]
apic_ipi:               total cycles 375671259, avg 3756
    -->These lines show the average cycles of each kvm_send_ipi_mask: 
3756

apic_ipi:       IPI[flat_send_IPI_mask] from CPU[0] to CPU[1]
apic_ipi:               total cycles 221961822, avg 2219
    -->These lines show the average cycles of each 
flat_send_IPI_mask: 2219

Just single target IPI is not eough.

     Wanpeng

Benchmark smp_call_function_single 
(https://github.com/bytedance/kvm-utils/blob/master/microbenchmark/ipi-bench/ipi_bench.c): 


  Test env:
  CPU: AMD EPYC 7642 48-Core Processor

  Kmod args(enable avic and disable nested):
  modprobe kvm-amd nested=0 avic=1 npt=1

  QEMU args(disable x2apic):
  ... -cpu host,x2apic=off ...

1> without no-kvm-pvipi:
ipi_bench_single wait[1], CPU0[NODE0] -> CPU1[NODE0], loop = 100000
      elapsed =        424945631 cycles, average =     4249 cycles
      ipitime =        385246136 cycles, average =     3852 cycles
ipi_bench_single wait[0], CPU0[NODE0] -> CPU1[NODE0], loop = 100000
      elapsed =        419057953 cycles, average =     4190 cycles

2> with no-kvm-pvipi:
ipi_bench_single wait[1], CPU0[NODE0] -> CPU1[NODE0], loop = 100000
      elapsed =        321756407 cycles, average =     3217 cycles
      ipitime =        299433550 cycles, average =     2994 cycles
ipi_bench_single wait[0], CPU0[NODE0] -> CPU1[NODE0], loop = 100000
      elapsed =        295382146 cycles, average =     2953 cycles

Hi, Wanpeng & Sean

Also benchmark redis(by 127.0.0.1) in a guest(2vCPU), 'no-kvm-pvipi' 
gets better performance.

Test env:
Host side: pin 2vCPU on 2core in a die.
Guest side: run command:
   taskset -c 1 ./redis-server --appendonly no
   taskset -c 0 ./redis-benchmark -h 127.0.0.1 -d 1024 -n 10000000 -t get

1> without no-kvm-pvipi:
redis QPS: 193203.12 requests per second
kvm_pv_send_ipi exit: ~18K/s

2> with no-kvm-pvipi:
redis QPS: 196028.47 requests per second
avic_incomplete_ipi_interception exit: ~5K/s

-- 
zhenwei pi